Assessment-driven cognition system

ABSTRACT

A computer-based assessment driven cognition system includes, in embodiments, a generator configured to present questions, where each question presented is associated with a stacking weight and has at least one correct response, and a question property evaluator configured to, for each question presented, determine a likelihood of randomly selecting the at least one correct response. The system also includes a question response receiver configured to receive at least one response to each question presented. Additionally, the system includes a question response evaluator configured to determine, for each question presented, when the at least one response is the at least one associated correct response, a likelihood that the at least one response is a false positive. The question response evaluator is further configured to adjust the stacking weight of each question presented depending at least upon the likelihood that the at least one response to the question is a false positive.

CROSS-REFERENCE TO RELATED APPLICATION

The present application claims priority under 35 U.S.C. § 119(e) toprovisional application No. 61/037,648 filed on Mar. 18, 2008, theentire contents of which are incorporated herein by reference.

TECHNICAL FIELD

The present invention relates to iterative learning and assessment tools(hereinafter “ILATs”). ILATs develop and assess a learner's knowledge byiterative means of evaluating the learner's responses to questions anddetermining whether and when any questions will be asked again to thelearner.

BACKGROUND

ILATs provide learners the opportunity to assess their knowledge andimprove their skills. Examples of learners who use ILATs include, butare not limited to, students at universities and employees at a company.ILATs operate to select the next question (or set of questions, orinstructional content block) to display to the learner.

However, current ILATs are limited in the ability to assess if thelearner selects a correct response to a question without knowledge thatthe response is correct.

SUMMARY

Embodiments of the present invention calculate a set of values for anILAT to help the ILAT determine which question, question set, orinstructional content to present next to a user (e.g., learner).Embodiments of the invention may be employed by new or existingassessment tools to help them determine a course of action resultingfrom the user's response to a problem. In general, it is contemplatedthat various embodiments of the present invention include assessmenttools working with existing ILATs and new ILATs containing new, integralfunctionalities.

Embodiments use a collection of operands to determine a likelihood thatthe user's correct response to a question is a false positive. Forexample, one or more operands may indicate a percent chance that theuser randomly selects the correct response to the question. In addition,one or more operands may indicate how many times the user responds tothe question incorrectly after answering the question correctly, howmany consecutive times the question is answered correctly orincorrectly, and/or how quickly the user correctly responds to thequestion compared to other user responses to the question. Further, oneor more operands may indicate the likelihood of selecting one or moreincorrect options to the question.

Embodiments accept the user's response to a question, use that responseand operands discussed above to determine a set of values including thelikelihood that the user's correct response is a false positive. Thosevalues can then be used to determine which question to present next.

In embodiments, a computer-based system for assessment driven learningincludes a generator configured to present questions received from aprocessor to a question presentation device utilizing a question moduleand a question database having a plurality of questions. The pluralityof questions are associated with a stacking weight and are orderedaccording to the stacking weight of each question. Each questionpresented includes at least one answer option and at least oneassociated correct response. The computer-based system includes aquestion property evaluator configured to utilize the question module todetermine a likelihood of randomly selecting the at least one associatedcorrect response. The question property evaluator considers, for eachquestion presented, at least the number of answer options and a numberof answer options a user is instructed to select. The computer-basedsystem includes a question response receiver configured to receive, froma device associated with the user, at least one response to eachquestion presented utilizing a communications module. The system furtherincludes a question response evaluator configured to, utilizing anevaluation module and a user response history database for each questionpresented, determine, when the at least one response is the at least oneassociated correct response, a likelihood that the at least one responseis a false positive using the question property evaluator, a responsehistory of the user, and a response history of one or more other users.The question response evaluator is further configured to adjust thestacking weight of each question presented within the question databasedepending at least upon the likelihood that the at least one response tothe question is a false positive.

While multiple embodiments are disclosed, still other embodiments of thepresent invention will become apparent to those skilled in the art fromthe following detailed description, which shows and describesillustrative embodiments of the invention. Accordingly, the drawings anddetailed description are to be regarded as illustrative in nature andnot restrictive.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an example ILAT process;

FIG. 2 illustrates an example configuration between a learning center,an organization, a computer, and a personal digital assistant;

FIG. 3 illustrates an example system including one or more libraries andone or more databases;

FIG. 4 illustrates an example system including one or more modules;

FIG. 5 illustrates an example set of questions;

FIG. 6 illustrates example properties of a question;

FIG. 7 illustrates users' histories of responding to a question;

FIG. 8 illustrates how each response by a user affects U_((cbi));

FIG. 9 illustrates an example process for calculating Q_((cbi));

FIG. 10 illustrates example response histories and a correspondingQ_((cbi)); Q_((pir));

FIG. 11 illustrates how UQ_((rca)) correlates with UQ_((ctc));

FIG. 12 illustrates users' response histories for a question and acorresponding UQ_((cbi)) and UQ_((ctc));

FIG. 13 illustrates an example process for calculating UQ_((ctc));

FIG. 14 illustrates when an option for a question is displayed and whena user responds to the question;

FIG. 15 illustrates how long it took for a user to respond to a questioncorrectly or incorrectly;

FIG. 16 illustrates an example property list for a question;

FIG. 17 illustrates an example distribution of correct responses to aquestion for one or more users;

FIG. 18 illustrates the response times illustrated in FIG. 17 and thecorresponding Q_((svac)) and Q_((scsd));

FIG. 19 illustrates a chart for UQ_((fpp1)) with correspondingUQ_((scsdfm)) and UQ_((ctc)) values;

FIG. 20 illustrates how a correct response is flagged as a falsepositive after a question is answered incorrectly;

FIG. 21 illustrates a chart for UQ_((fpp2)) with correspondingUQ_((tgf)) and UQ_((ctc)) values;

FIG. 22 illustrates two example mathematical questions;

FIG. 23 illustrates a question with corresponding answer options and acorresponding Q_((qod));

FIG. 24 illustrates how the value of Q_((qod)) changes as thedistribution of answer option selections varies;

FIGS. 25A and 25B illustrate how changes in values of operands used inthe Q_((qod)) equation for a question affect Q_((qod));

FIG. 26 illustrates an example process for calculating Q_((qod));

FIG. 27 illustrates an example chart for Q_((fpo)) with correspondingQ_((qod)) and Q_((pre)) values;

FIG. 28 illustrates an example chart for UQ_((fpp3)) with correspondingQ_((fpo)) and UQ_((ctc)) values;

FIG. 29 illustrates a question illustrated in submit mode and certaintymode;

FIG. 30 illustrates an example process for displaying a question in thecertainty mode;

FIG. 31 illustrates a chart for UQ_((cw)) with corresponding UQ_((c))and UQ_((ctc)) values with U_((ct)) set to 80%;

FIG. 32 illustrates a chart for UQ_((cw)) with corresponding UQ_((c))and UQ_((ctc)) values with U_((ct)) set to 50%;

FIG. 33 illustrates an example process for calculating UQ_((cw));

FIG. 34 illustrates an example process for calculating UQ_((fpp));

FIG. 35 illustrates an example process for calculating UQ_((sw));

FIG. 36 illustrates an example process for calculating a recommendedstacking weight and its constituent operands;

FIG. 37 illustrates one or more operands determined by the processillustrated in FIG. 36;

FIG. 38 illustrates an example properties list for a question;

FIG. 39 illustrates an example set of data of a user's response to aquestion;

FIG. 40 illustrates another example set of data of a user's response toa question;

FIG. 41 illustrates another set of example data of a user's response toa question;

FIGS. 42A to 42B illustrate a control panel that permits anadministrator to select operands;

FIG. 43 illustrates an example computing device; and

FIG. 44 illustrates an example parallel processing device.

DETAILED DESCRIPTION

The phrases “in some embodiments,” “according to some embodiments,” “inthe embodiments shown,” “in other embodiments,” and the like generallymean the particular feature, structure, or characteristic following thephrase is included in at least one embodiment of the present invention,and may be included in more than one embodiment of the presentinvention. In addition, such phrases do not necessarily refer to thesame embodiments or different embodiments.

In embodiments, the terms user, learner, and respondent are usedinterchangeably.

ILATs provide questions to users and evaluate responses to the questionsto determine if and when a user will see the question again. Inembodiments, questions presented to a user are displayed in aninterface. In other (or overlapping) embodiments, questions presented toa user are physical flashcards that the user may physically re-order ormove.

FIG. 1 illustrates an example ILAT process. In embodiments, the processstarts at 100 where for a given user, a question stack, a performancethreshold, and an instruction interval for the user can be assigned. Inembodiments, a question stack can be a subset of one or more questionsselected from a plurality of questions. An example instruction intervalincludes, but is not limited to, a specified number of questionspresented to the user. An example performance threshold includes, but isnot limited to, a percentage of questions the user is required to answercorrectly before completing the instruction interval.

ILAT process flow proceeds from 100 to 102 where a user receives auser-specific weighted question stack. In embodiments, a user-specificweighted question stack can be an ordered question stack where eachquestion in the question stack can be associated with a weight value. Asan example, a question with the lowest weight value compared to theweight values of the other questions in the question stack can beordered at the top of the question stack. The ILAT process flow proceedsfrom 102 to 104 to determine if the instruction interval is reached. Asan example, an instruction interval may specify that a user is notsuppose to answer more than 100 questions. Accordingly, when the useranswers the 100th question, the instruction interval is reached. If theinstruction interval is reached, the ILAT process flow proceeds from 104to 106 to instruct the user. An ILAT may provide a report indicatingwhich questions were answered correctly to instruct the user on theareas the user needs to improve.

If the instruction interval is not reached, the ILAT process flowproceeds from 104 to 108 to determine if the user's performancethreshold is reached. As an example, if the performance threshold is setto 50%, and the user has already answered 10 questions, the performancethreshold is reached when the user has answered 5 questions correctly.If the user's performance threshold is reached, the ILAT process ends.If the user's performance threshold 108 is not reached, the ILAT processflow proceeds from 108 to 110 to select the next question to display tothe user. As an example, the question at the top of the stack (e.g., thequestion with the lowest weight value) is selected to display to theuser.

The ILAT process flow proceeds from 110 to 112 where the question isdisplayed to a user and the ILAT waits for the user response. Inembodiments, questions are displayed to the user using any desiredformat. For example, if the user belongs to a university, the format ofthe question display may include colors and logos associated with theuniversity. ILAT process flow proceeds from 112 to 114 to determine ifthe user responded to the displayed question. If the user did notrespond to the displayed question, the ILAT process ends. As an example,the ILAT process may time out after not receiving a response for apredetermined period of time.

If the user responded to the displayed question, the ILAT process flowproceeds from 114 to 116 to calculate the displayed question's newstacking weight. The ILAT process flow proceeds from 116 to 118 toreturn the newly weighted question to the question stack. As an example,when the question's new stacking weight is calculated, the question canbe reordered in the question stack according to the new stacking weight.The ILAT process flow returns from 118 to 102 where the processdescribed above for 102 is repeated.

ILATs that implement the process illustrated in FIG. 1 may evaluatewhether a response is correct or incorrect in determining the ILAT'snext course of action. Embodiments operate on the assertion that it isinsufficient to simply evaluate whether a response to a question isright or wrong for all questions in the stack. Consider, for example, anILAT in which questions are always presented with a single correctresponse option and two incorrect response options (i.e. distracters).Without considering the actual content of the response options, thepossibility of selecting the one correct response by random chance isalways 33%. In embodiments, the percent random chance of selecting acorrect response to a question as a variable is referred to asQ_((prc)). If the learner can determine that one of the two distractersis incorrect, then the question's Q_((prc)) rises to 50%.

With a 33% chance of randomly selecting the correct answer, andcertainly with a 50% chance, the odds are too high that a correctresponse may not be a true reflection of the learner's knowledge of thequestion content. A correct response to a question that does not reflectthe learner's true knowledge is referred to as a false positive.Embodiments apply various techniques to identify false positives. Thesetechniques can be used by an ILAT to help the ILAT determine whether andwhen to re-present a question to a learner. Embodiments calculate anduse the Q_((prc)) variable in evaluating the likelihood of a falsepositive.

Embodiments make use of a learner's reported level of certainty in thelearner's response to a question to determine if a correct response to aquestion is a false positive. As an example, if a learner reports a lowlevel of certainty while responding to the question correctly, then thecorrect response is treated as a false positive. In addition to usingthe learner's reported level of certainty, embodiments use anothervariable referred to as a Certainty Threshold. As an example, if thelevel of reported certainty (which can vary from one embodiment toanother) is below the Certainty Threshold, any correct response isregarded as a false positive. Embodiments can be run in Certainty Mode.When run in Certainty Mode, the embodiment asks the learner to reporthis level of certainty. When not run in Certainty Mode, the learner isnot asked to report on his level of certainty.

Embodiments use a variable UQ_((fpp)) to represent a likelihood that aresponse to a question is a false positive. As an example, UQ_((fpp))can be associated with a scale of 0 to 1 to indicate a response'sprobability of being a false positive where a definite false positivehas a UQ_((fpp)) equal to 1. In embodiments, an incorrect responsealways has a UQ_((fpp)) equal to 0 since an incorrect response cannot bea false positive. A UQ_((fpp)) greater than 0 and less than 1 correlatesto the probability that a correct response is actually a false positive(e.g., the learner responded to the question correctly without knowledgeof the question's correct answer). In embodiments, this probability isnot calculated from a learner's reported level of certainty, but rathercan be calculated from other variables including at least Q_((prc)).

Embodiments use a variable UQ_((ctc)) to represent the number ofconsecutive correct responses or consecutive incorrect responses by alearner to a specific question that have occurred most recently in thelearner's history of responses to the question. The UQ_((ctc)) variableincreases or decreases the stacking weight of a question. A question'sstacking weight can be represented by the variable UQ_((sw)), where alow stacking weight may represent the likelihood that a question will bere-presented to the learner sooner compared to a high stacking weight.An ILAT may convert a low stacking weight to a high stacking weight, andvice versa, depending on how the ILAT chooses to use the stackingweight. If the user consistently answers a question incorrectly,UQ_((ctc)) has a negative value and the question's stacking weight maydecrease since the user needs to see the question again soon. If theuser consistently answers a question correctly, UQ_((ctc)) has apositive value and the question's stacking weight may increase since theuser does not need to see the question again soon. The ILAT may useUQ_((ctc)) as the sole operand in calculating UQ_((sw)) for an incorrectresponse to a question. In another example, for a correct response to aquestion, the ILAT may calculate UQ_((sw)) using UQ_((ctc)) andQ_((prc)). Accordingly, in embodiments, the ILAT has discretion overwhich operands to use in determining if a question is presented againsooner or later.

In embodiments, any mechanism or device configured to determine one ormore operands to determine UQ_((fpp)) and/or UQ_((sw)) may be locatedexternal to the ILAT. In other (or overlapping) embodiments, anymechanism or device configured to determine one or more operands todetermine UQ_((fpp)) and/or UQ_((sw)) may be incorporated as part of theILAT itself. Accordingly, any embodiment describing returning values toan ILAT may include any mechanism or device located externally to theILAT or as part of the ILAT itself.

In embodiments, questions are presented to users via an online system.FIG. 2 illustrates an example configuration between a learning center200, an organization 204, a computer 206, and a personal digitalassistant (PDA) 210. In embodiments, the learning center includes one ormore servers that performs the ILAT process illustrated in FIG. 1. Forexample, the organization 204 can be any school, university, or company,and the computer 206 can be any desired computing device such as, butnot limited to, a personal desktop located at a residential address. Inaddition, the PDA can be any desired digital assistant such as aBlackberry.

In embodiments, the learning center 200 communicates with theorganization 204 and computer 206 over a public network such as theInternet. As an illustrative example, the organization 204 can be auniversity where students at the university may register for coursesrequiring students to take online tests. Accordingly, when students aretaking online tests at the university, one or more questions may bedownloaded from the learning center 200 to a terminal located at theuniversity over the public network 202. In another example, if studentsare permitted to take online tests while at home, students may downloadone or more questions from the learning center 200 to their homecomputer 206 over the public network 202. In yet another example,students may be permitted to take online tests associated with theircourses using their own PDA. For example, a student may download one ormore questions from the learning center 200 to their PDA 210 over amobile network 208.

FIG. 3 illustrates an example system 300 including one or more librariesand one or more databases. As an example, the system 300 is incorporatedin the learning center 200 (FIG. 2), though aspects of system 300 canalso be incorporated into other portions of the present invention. Inembodiments, the system 300 includes a communications library 302, aregistration database 304, a course database 306, a questions database308, and a user history response database 310.

In embodiments, the communications library 302 contains information forcommunicating with a network using any desired network protocol. As anexample, if a network is using the Global System for Mobile (GSM)communications protocol, the communications library 302 can include allinformation used for communicating with that particular network usingthe GSM protocol. If a network is using the IP protocol, thecommunications library 302 includes all information used forcommunication with that particular network using the IP protocol.

In embodiments, the registration database 304 can include registrationprofiles of users who register for online courses. For example, aregistration profile for a particular user may indicate each course thata user is taking. In embodiments, the course database 306 can includeinformation associated with courses provided by the learning center 200.The course database 306 may include display formats for each course. Forexample, if a course is offered for a particular university, a displayformat for that course may include colors and logos associated with theuniversity. The course database 306 may also include schedules for eachcourse. As an example, a schedule associated with a course may indicatewhat day of the week the course is offered and when tests for the courseare provided.

In embodiments, the questions database 308 stores each question providedby the learning center 200 (FIG. 2) and the properties of each question.As an example, a property of a question includes, but is not limited to,the total number of times the question has been answered, the totalnumber of times the question has been answered correctly, and the totalnumber of times the question has been answered incorrectly. Eachquestion stored in the database may include at least one answer optionsand at least one associated correct response. In other (or overlapping)embodiments, the questions database 308 only stores the properties ofeach question instead of the actual questions.

In embodiments, the user history response database 310 stores a responsehistory for each user. As an example, a response history for aparticular question may indicate when the user answered the questioncorrectly and when the user answered the question incorrectly.

The term “module” refers broadly to a software, hardware, or firmwarecomponent (or any combination thereof). Modules are typically functionalcomponents that can generate useful data or other output using specifiedinput(s). A module may or may not be self-contained. An applicationprogram (also called an “application”) may include one or more modules,and/or a module can include one or more application programs.

FIG. 4 illustrates an example system 400 including one or more modules.The system 400 may be incorporated into the learning center 200 (FIG.2), though aspects of system 400 can also be incorporated into otherportions of the present invention. In embodiments, the system 400includes a communications module 402, a registration module 404, acourse module 406, a question module 408, an evaluation module 410, astack update module 412, and a tuner module 414.

In embodiments, the communications module 402 may be configured tocommunicate with each network and/or device in communication with thelearning center 200 (FIG. 2). As an example, if the learning system 300is in communication with a device over a network using the IP protocol,the communications module 402 accesses the communications library 302(FIG. 3) to determine how to send and receive data using the IPprotocol. Accordingly, the communications module can utilize thecommunications library 302 (FIG. 3) to send and receive data using anynumber of desired communication protocols. In embodiments, thecommunications module 402 operates as a question response receiver.

In embodiments, the registration module 404 may perform a registrationprocess to register users for courses. As an example, when a userregisters for a course, the registration module 404 creates aregistration profile for that user and stores the profile in theregistration database 304. In embodiments, the course module 406executes the courses for each registered user. For example, when acourse is scheduled to be provided to users on a particular day, thecourse module 406 retrieves information associated with the course fromthe course database 306 and provides the course to the registered users.As another example, when a test associated with the course is scheduledto be provided on a particular day, the course module 406 retrieves thequestions associated with the test from the question database 308 andprovides the test to the users.

In embodiments, the question module 408 displays questions to users,receives responses from the users, updates the question properties ofthe question, and updates the user's response history. For example, uponreceiving a response from the user, the question module 408 updates theproperties for the question, such as the total number of times aquestion is answered, and stores the updated properties in the questiondatabase 308 (FIG. 3). As another example, upon receiving a responsefrom the user, the question module 408 updates the user responsehistory, such as whether the current response is correct, and stores theupdated user response history in the user history response database 310.In embodiments, a generator (also called) a “question generator”utilizes the question module 408 and the question database 308 topresent questions to a question presentation device. In someembodiments, the question generator and question module are one and thesame. In embodiments, a question presentation device can be any desireddevice suitable for presenting one or more questions to a user. Examplesof question presentation devices include, but are not limited to, LCDdisplay screens or any audio system configured to play an audio filecontaining the question to a user and receive a voice response from theuser. In further embodiments, a question property evaluator utilizes thequestion module 408 to determine one or more properties associated witha question.

In embodiments, the evaluation module 410 receives a response to aquestion from the question module 408 and determines a likelihood thatthe response is a false positive (UQ_((fpp))) and a new stacking weight(UQ_((sw))) for the question using one or more operands such as a user'sreported level of certainty and/or UQ_((fpp)). In embodiments, aquestion response evaluator can utilize the evaluation module 410 (or isthe evaluation module 410) to determine one or more operands associatedwith a user's response to a question. In embodiments, the stack updatemodule 412 updates a question stack using the results of the evaluationmodule 410. In embodiments, the tuner module 414 permits adjustment ofone or more operands used by the evaluation module 410 to determineUQ_((fpp)) and UQ_((sw)).

Calculating Q_((prc))

As discussed above, the Question Percent Random Chance (Q_((prc)))represents the percent chance of randomly selecting the correctanswer(s) to a question. In embodiments, whenever a new question iscreated, the question module 408 (FIG. 4) automatically calculates thequestion's Q_((prc)) and stores this value as a property of the questionin the question database 308 (FIG. 3). In other (or overlapping)embodiments, whenever a new question is created, the question'sQ_((prc)) can be manually entered by the author of the question andstored as a property of the question in the question database 308 (FIG.3).

FIG. 5 illustrates a question set of eight questions. As illustrated inFIG. 5, each question has a number of answer options. The responseoptions are located in the cells where questions and answer optionsintersect and contain information about the correctness of thecorresponding response option. For example, the number 1 symbolizes aresponse option which is a correct solution to the problem. Further, thenumber 0 symbolizes a distracter (i.e., incorrect response option). Inembodiments, the Q_((to)) (Total Options) is a count of the number ofanswer options associated with a question. For example, the Q_((to)) forQuestion 3 is 4 since Question 3 has 4 answer options.

In embodiments, the Q_((s1)) (Selection Limit) denotes the number ofoptions that a respondent is instructed or permitted to select. Forexample, if Question 1 is a True/False question type, the user would beinstructed to select only one option. Therefore, Q_((s1)) is equal to 1for question 1. In another example, Question 2 has only two options, butthe user may be instructed to “Select all that apply.” Accordingly,Q_((s1)) is equal to 2 for Question 2.

The Q_((prc)) row in FIG. 5 denotes the percent chance of randomlyselecting the correct answer, expressed as a decimal. In embodiments,Q_((prc)) can be derived using the following calculation where ^(Q(to))C_(Q(s1)) represents the number of possible combinations from Q_((to))taking Q_((s1)) at a time:

${IF}\mspace{14mu} \left( {Q_{({to})} > Q_{({sl})}} \right)\mspace{14mu} {THEN}\left\{ {Q_{({prc})} = {\frac{1}{{}_{}^{Q({to})}{}_{Q({sl})}^{}}\left( Q_{({sl})} \right)}} \right\} \mspace{14mu} {ELSE}\mspace{14mu} \left\{ {Q_{({prc})} = \frac{1}{\left( 2^{Q_{({to})}} \right)}} \right\}$

Using the above equation and Question 2 in FIG. 5 as an example,Question 2 has 2 options (Q_((to))=2) where the user is instructed toselect all options that apply for the question (Q_((s1))=2).Accordingly, the possible choices are 00, 01, 10, and 11 where 00denotes the possibility that none of the answer options are selected.Since FIG. 5 indicates 10 as the correct response out of a possible 4four responses, the probability of randomly selecting the correctresponse (Q_((prc))) is ¼=0.25, which is verified by the above equation.

In embodiments, some question types may not permit the automatedcalculation of Q_((prc)). An example of such a question type may be afill-in-the-blank exercise in which the respondent must type a word orphrase as opposed to selecting the correct answer from a list. In thisexample, the percent chance of producing a specific written word atrandom may be relatively low. In this case, the question author mightdecide to manually assign the question a Q_((prc)) of 0.01.

Calculating Q_((ttc)) and Q_((tti))

When a question is selected by the ILAT and a respondent responds to thequestion, properties are added and/or updated for each of the followingby the question module 408: 1) the question; 2) the question's answeroptions; 3) the respondent's interaction with the question; and 4) therespondent. In embodiments, the variable Q_((ttc)) (Question Total TimesCorrect) is the total number of times that a correct response has beensubmitted for a question. The variable Q(tti) (Question Total TimesIncorrect) is the total number of times that an incorrect response hasbeen submitted for a question.

FIG. 6 shows how, in embodiments, the properties of Question 3 and itsoptions can be updated when a respondent responds to a question. Asillustrated in FIG. 6, the variables Q_((to)), Q_((s1)), and Q_((prc))are provided as properties of Question 3. In embodiments, the variableQO_((c)) indicates if an answer option is correct (1) or incorrect (0).For example, since option 1 of Question 3 is correct, the QO_((c)) forthis option is 1. In embodiments, the variable QO_((ts)) indicates thetotal number of times that an answer option has been selected. As anexample, the first time a response is submitted for Question 3, theresponse selected is Option 2, which is an incorrect response.Therefore, the total number of selections for Question 3, Option 2 isequal to one (e.g., Q₃O_(2(ts))=1). Furthermore, the total number oftimes that Q₃ has been answered correctly is 0 (e.g., Q_(3(ttc))=0).Additionally, the total number of times that Q₃ has been answeredincorrectly is 1 (e.g., Q_(3(tti))=1). As discussed above, the questionmodule 408 (FIG. 4) may receive responses to questions, update theproperties of the questions accordingly, and store the updatedproperties in the question database 308 (FIG. 3). Accordingly, theproperties for Question 3 as illustrated in FIG. 6 may be retrieved fromthe question database 308 (FIG. 3) when determining if a correctresponse to Question 3 is a false positive or readjusting the stackingweight for Question 3.

Calculating UQ_((cbi)) and Q_((cbi))

If a respondent gets an answer wrong after having previously gotten itright, a likely cause may be either (a) the respondent forgot thecorrect answer, or (b) the respondent simply guessed correctly whenpreviously answering the question. The latter case (b) is considered asa false positive. The former case (a) is treated as a false positivebecause previous correct responses to the question are no longerrepresentative of the respondent's knowledge. The variable UQ_((cbi))(User-Question Correct Before Incorrect), in embodiments, is a count ofthe number of times that a respondent correctly responds to a problemprior to incorrectly responding to that problem. In embodiments, thevariable Q_((cbi)) (Question Correct Before Incorrect) is the sum of allUQ_((cbi)) for a given question. In embodiments, the question module 408determines UQ_((cbi)) for each user and stores the value in the userhistory response database 310 (FIG. 3). The question module 408 (FIG. 4)determines Q_((cbi)) for each question and stores the value in thequestion database 308 (FIG. 3).

FIG. 7 illustrates UQ_((cbi)) with respect to users' histories ofresponding to a question. The “UserID” column in FIG. 7 identifies aparticular user. The “QuestionID” column in FIG. 7 identifies aparticular question. The “UQ(tti)” column in FIG. 7 indicates the totalnumber of times the user answered the question identified in the“QuestionID” column incorrectly.

The “Response Correctness Array” in FIG. 7 indicates each users' historyfor responding to the question identified in the “QuestionID.” Inembodiments, a 0 designates an incorrect response and a 1 designates acorrect response. FIG. 7 illustrates that User 5 is responding toQuestion 1 erratically where User 5 has answered Question 1 incorrectlytwice after responding to Question 1 correctly. FIG. 8 illustrates howeach response by User 5 to Question 1 affects U₅Q_(1(cbi)). Inembodiments, UQ_((cbi)) is augmented by a number of correct responsesbefore a question is answered incorrectly. For example, the first timeUser 5 answers Question 1 incorrectly after responding to Question 1correctly, U₅Q_(1(cbi)) is incremented by 1 since there is one correctresponse before Question 1 is answered incorrectly. The second time User5 answers Question 1 incorrectly after responding to Q₁ correctly,U₅Q_(1(cbi)) is incremented by 2 since there are two correct responsesbefore Question 1 is answered incorrectly again. Accordingly, asillustrated in FIG. 7, the UQ_((cbi)) for User 5 is 3. As anotherexample, FIG. 7 illustrates that User 2 answered Question 1 incorrectlyafter answering Question 1 correctly once. Accordingly, the UQ_((cbi))for User 2 is 1.

In embodiments, the variable UQ_((cbi)) is associated with the variableU_((tgf)) (User Tendency to Guess or Forget), which represents alikelihood that when a user is answering a question, the user isguessing. U_((tgf)) may be calculated by the following equation:

${U_{({tgf})} = \frac{U_{({gfc})}}{U_{({tqa})}}},$

where U_((gfc)) represents a user's guess or forget count and U_((tqa))represents the total questions answered by the user. U_((gfc)) may beequal to a user's UQ_((cbi)) for all questions answered by the usersince an increase in UQ_((cbi)) is an indication that the user isguessing the answers to a question. U_((tqa)) may be incremented by 1anytime a user submits a response to any question.

FIG. 9 illustrates an example process for calculating Q_((cbi)) for aparticular question. The process starts at 900 where the UQ_((cbi)) fora particular user answering a particular question is updated (e.g.,UQ_((cbi)) is either incremented or kept the same). Process flowproceeds from 900 to 902 to determine if UQ_((cbi)) increased from aprevious UQ_((cbi)) value. As discussed above and illustrated in FIG. 7,UQ_((cbi)) for a particular user is incremented whenever a user answersa question incorrectly after answering the question correctly. IfUQ_((cbi)) did increase from a previous UQ_((cbi)) value, process flowproceeds from 902 to 904 to increase U_((gfc)) by the same amount thatUQ_((cbi)) increased. As discussed above, U_((gfc)) is equal to a user'sUQ_((cbi)) for all questions answered by the user since an increase inUQ_((cbi)) is an indication that the user is guessing the answers to aquestion. If UQ_((cbi)) did not increase from a previous UQ_((cbi))value, process flow proceeds from 902 to 908 to calculate Q_((cbi)).

Process flow proceeds from 906 to 908 to calculate Q_((cbi)) for aparticular question. In embodiments Q_((cbi)) is the sum of UQ_((cbi))for each user for a particular question. For example, referring to FIG.7, the sum of UQ_((cbi)) for Question 1 for each user is 5.

Calculating Q(pir)

In embodiments, the variable Q_((pir)) (Question Percentage of IncorrectResponses) is the percentage, expressed as a decimal, of the totalnumber of users' responses that are incorrect or considered falsepositives. The Q_((pir)) for a particular question expressed as adecimal can be calculated by the following equation:

$Q_{({pir})} = \frac{\left( {Q_{({tti})} + Q_{({cbi})}} \right)}{\left( {Q_{({ttc})} + Q_{({tti})}} \right)}$

As illustrated in the above equation, Q_((cbi)) offsets Q_((tti)) whencalculating Q_((pir)). Particularly, Q_((tti)) is increased by Q_((cbi))since Q_((cbi)) indicates that previous correct responses were falsepositives (e.g., the user answered the question correctly withoutknowledge of the correct answer). Thus, as illustrated in the aboveequation, the percentage of questions treated as incorrect responsesincreases as Q_((cbi)) increases. In embodiments, the question module408 (FIG. 3) determines Q_((pir)) and stores the value in the questiondatabase (FIG. 308). In other (or overlapping) embodiments, Q_((pir)) isnot stored as a value since the question module 408 may calculateQ_((pir)) in real time.

FIG. 10 illustrates respondent response histories and a correspondingQ_((cbi)) and Q_((pir)) for Questions 1-4. As illustrated in FIG. 10,the Q_((cbi)) for Question 1 is 2 since there were two occurrences ofanswering a question correctly before answering the questionincorrectly. Accordingly, the Q_((pir)) for Question 1 may be calculatedusing the equation above and the Q_((cbi)), Q_((tti)), and Q_((ttc))values illustrated in FIG. 10 for Question 1. As also illustrated inFIG. 10, responses to Question 2 appear to contain the highest number offalse positives since the Q_((cbi)) for Question 2 is higher than anyother question. In embodiments, the question module 408 (FIG. 4)determines Q_((pir)) and stores the value in the question database 308(FIG. 3).

Calculating UQ(ctc)

In embodiments, the variable UQ_((ctc)) (User-Question Consecutive TimesCorrect) is the number of times that a Response Correctness Arrayterminates with one or more consecutive correct responses (in which caseUQ_((ctc)) can be expressed as a positive integer), or one or moreconsecutive incorrect responses (in which case UQ_((ctc)) can beexpressed as a negative integer). If a Response Correctness Array isempty, then UQ_((ctc)) has a value of zero. In embodiments, theUQ_((ctc)) variable is used to determine the likelihood that a correctresponse to a question is a false positive.

As discussed above, the chronological history of correctness of a user'sset of responses to a question can be indicated in a question's responsecorrectness array, or UQ_((rca)). In embodiments, correct and incorrectresponses may be recorded in a UQ_((rca)) using a series of 0s (torepresent a user's incorrect responses to the question) and 1s (torepresent a user's correct responses to the question). The UQ_((rca))may be stored in the user history response database 310 (FIG. 3). As anexample, a UQ_((rca)) might have the following sequence: 0,1,0,1,1. Inthis example, the UQ_((rca)) indicates that the user responded fivetimes to the question, and UQ_((ctc)) has a value of 2 since the userhas responded to the question correctly two times consecutively. FIG. 11illustrates how UQ_((rca)) correlates with UQ_((ctc)). As illustrated inthe first row in FIG. 11, the default value for UQ_((ctc) is) 0 whenthere are no responses. As further illustrated in the fifth row ofresponses in FIG. 11, UQ_((ctc)) is −1 since the user's most recentresponse is incorrect after answering the question correctly in theprevious response.

FIG. 12, illustrates users' response histories for Question 1 and acorresponding UQ_((cbi)) and UQ_((ctc)) for each user. For example, User2 has a UQ_((rca)) of 0,1,0,1. Accordingly, since User 2 answeredQuestion 1 correctly once prior to answering Question 1 incorrectly,UQ_((cbi)) for User 2 for Question 1 is 1. Additionally, since User 2'smost recent response to Question 1 is correct after answering thequestion incorrectly in the previous response, UQ_((ctc)) for User 2 is1.

FIG. 13 illustrates an example process for calculating UQ_((ctc)). Inembodiments, the question module 408 performs the process illustrated inFIG. 13. The process starts at 1300 to determine the current value ofUQ_((ctc)). If the current value of UQ_((ctc)) is greater than or equalto 0, process flow proceeds from 1302 to determine if the user's currentresponse to the question is correct. If the user's current response tothe question is incorrect, process flow proceeds from 1302 to 1304 tochange UQ_((ctc)) to −1. If the user's current response to the questionis correct, process flow proceeds from 1302 to 1305 to incrementUQ_((ctc)) by 1. Process flow proceeds from 1304 and 1305 to 1306 tostore UQ_((ctc)). The variable UQ_((ctc)) for each user may be stored inthe user history response database 310 (FIG. 3).

If the current value of UQ_((ctc)) is less than 0 at 1300, process flowproceeds from 1300 to 1308 to determine if the user's current responseto a question is correct. If the user's current response to the questionis incorrect, process flow proceeds from 1308 to 1310 to decrementUQ_((ctc)) by 1. If the user's current response to the question iscorrect, process flow proceeds from 1308 to 1312 to change UQ_((ctc))to 1. Process flow proceeds from 1310 and 1312 to 1306 to storeUQ_((ctc)).

Calculating Q_((scav)) and Q_((scsd))

In embodiments, the variable UQ_((dt)) (User-Question Display Time)represents the time when a question is displayed to a respondent, andthe variable UQ_((rt)) (User-Question Response Time) represents the timewhen a respondent responds to a question. FIG. 14 illustrates when anoption for a question is displayed and when a user responds to thequestion. The “tDisplayed” column of FIG. 14 illustrates when the optionfor the question is displayed. The “tResponded” column of FIG. 14illustrates when the user responds to the question. For example, FIG. 14illustrates that Option 1 of Question 9 was displayed to User 1 on Sep.10, 2008 at 9:58:37, and User 1 submitted a response to Question 9 onthe same day 12 seconds later.

The “Response” column in FIG. 14 indicates the type of responsesubmitted for the question. For example, a 0 in the “Response” columnindicates that an option is not selected, and a 1 in the “Response”column indicates that the option is selected. The “Response Correctness”column in FIG. 14 indicates if the user's response is correct. Forexample, a 0 in the “Response Correctness” column indicates that theuser's response is incorrect, and a 1 in the “Response Correctness”column indicates that the user's response is correct. As illustrated inFIG. 14, the first time (id₍₁₎) Option 1 for Question 9 is displayed toUser 1, the user did not select this option, which is an incorrectresponse. The second time (id₍₅₎) Option 1 for Question 9 is displayedto User 1, the user selected this option, which is a correct response.

In embodiments, UQ_((sc)) (User-Question Seconds to Correct) representsthe number of seconds it took for a user to submit a correct answer to aquestion. In embodiments, UQ_((si)) (User-Question Seconds to Incorrect)represents the number of seconds it took for the user to submit anincorrect answer to a question. FIG. 15 illustrates how long it took fora user to respond to a question correctly or incorrectly. The “id”column in FIG. 15 represents an order of responses received by an ILATfor any particular question. For example, FIG. 15 illustrates that anILAT's first received response is a response from User 1 for Question 9,and the ILAT's seventh received response is a response from User 4 forQuestion 9. The “Seconds to Correct” column in FIG. 15 corresponds tothe UQ_((sc)), and the “Seconds to Incorrect” column corresponds toUQ_((si)). As illustrated in FIG. 15, the first time User 1 responded toQuestion 9, User 1 responded incorrectly in 12 seconds (e.g.,UQ_((si))=12). The second time User responded to Question 9, Userresponded correctly in 9 seconds (e.g., UQ_((sc))=9).

In embodiments, upon updating a UQ_((sc)) or UQ_((si)) value, if morethan one UQ_((sc)) or UQ_((si)) exists for a question, the averagenumber of seconds to provide a correct response (Q_((scav))) orincorrect response (Q_((siav))) can be updated by the question module408 (FIG. 4) as a property of the question. In embodiments, Q_((scav))equals the total of all UQ_((sc)) for that question divided by thenumber of instances of UQ_((sc)) (e.g., the number of times the questionhas been answered correctly). In embodiments, Q_((siav)) equals thetotal of all UQ_((si)) for that Q divided by the number of instances ofUQ_((si)) (e.g., the number of times the question has been answeredincorrectly). Furthermore, embodiments may record standard deviationsfor correct (Q_((scsd))=STDEVP (all UQ_((sc)))) and incorrect(Q_((sisd)))=STDEVP (all UQ_((si)))) response times. The function STDEVPmay include any desired method for calculating a standard deviationusing any desired probability distribution. In embodiments, the STDEVPfunction can be performed using a methodology similar to the methodologyperformed by Microsoft Excel for the STDEVP function.

FIG. 16 illustrates an example property list for Question 9. Asillustrated in FIG. 16, the average number of seconds to provide acorrect response (Q_((scav))) to Question 9 is 12.5 seconds, while theaverage number of seconds to provide an incorrect response (Q_((siav)))to Question 9 is 16 seconds. The standard deviation for responding toQuestion 9 correctly is 2.86 seconds, while the standard deviation forresponding to Question 9 incorrectly is 3.3.

The Effect of a User's Response Time on UQ_((fpp))

In embodiments, UQ_((sc)) is used to determine the probability that auser's correct response to a question is a false positive. As anexample, the likelihood that a response is a false positive is inverselyproportional to the respondent's correct response time when that time islower than the average response time to answer a question correctly(i.e., Q_((scav))). Consider the following example: The average amountof time in which a correct response is given for Question 1 is 26seconds. User 3 (who spent only two seconds to provide a correctresponse) is substantially more likely than User 2 (who spent 30 secondsto provide a correct response) to have guessed at the answer since thefast response time may indicate that User 3 is randomly selecting aresponse to a question without consideration of the content of all theresponses to the question.

FIG. 17 illustrates an example distribution of correct responses toQuestion 1 for Users 1-7. In the example discussed above, User 3 hasUQ_((sc))=2 and User 2 has a UQ_((sc))=30. In embodiments, the variableUQ_((scse)) represents a standard error from the standard deviationwhere UQ_((scse))=STANDARDIZE(UQ_((sc)), Q_((scav)), Q_((scsd))). Usingthis equation for UQ_((scse)) and the distribution of responsesillustrated in FIG. 17, a UQ_((sc))=2 gives us a standard error(UQ_((scse))) of −1.185216. Using a rounding calculation ofUQ_((scsdfm))=CEILING(ABS(MIN(UQ_((scse)),0)),1), UQ_((scse)) is roundedup to indicate that a response time of two seconds, according to thedistribution of FIG. 17 is two standard deviations from the mean. Inembodiments, the STANDARDIZE and CEILING functions are performed usingmethodologies similar to the methodologies performed by Microsoft Excelfor the STANDARDIZE and CEILING functions.

As indicated in the above calculation for UQ_((scsdfm)), any responsetime higher than the mean generates UQ_((scsdfm))=0. That is, accordingto embodiments, a response time which is higher than the averageresponse time to answer the question correctly has no effect on theresponse's false positive probability (UQ_((fpp))). FIG. 18 illustratesthe response times depicted in FIG. 17 and the corresponding Q_((scav))and Q_((scsd)) for this distribution. FIG. 18 also illustrates theQ_((scse)) for User 3.

In embodiments, the first operand in the UQ_((fpp)) calculationUQ_((fpp1)), which may be calculated by the evaluation module 410 (FIG.4). UQ_((fpp1)) may be calculated according to the following equation:

${IF}\mspace{14mu} \left( {{UQ}_{({scsdfm})} > 0} \right)\mspace{14mu} {THEN}\mspace{14mu} \left\{ {{UQ}_{({{fpp}\; 1})}+=\frac{{UQ}_{({scsdfm})}}{{UQ}_{({ctc})}^{{UQ}_{({ctc})}}*\left( {5 + {UQ}_{({scsdfm})}} \right)}} \right\}$

According to the above equation, and as illustrated in FIG. 19,UQ_((fpp1)) increases as UQ_((scsdfm)) increases, but decreases asUQ_((ctc)) increases. UQ_((fpp1)) increases as UQ_((scsdfm)) increasesbecause a response time that is faster than the average response timeindicates that a user may be randomly selecting a response. However,UQ_((fpp1)) decreases as UQ_((ctc)) increases because if a user isconsistently selecting the correct response for a question, the user isless likely randomly selecting the correct response. As illustrated inFIG. 19, one consecutive correct response that is nominally faster(UQ_((scsdfm))=1) than average increases the false positive probability(U_((fpp1))) by 0.17. One consecutive correct response that is radicallyfaster than average (UQ_((scsdfm))=3) increases the false positiveprobability by 0.38. However, a second consecutive correct responseradically faster than average increases the false positive probabilityby only 0.09 (e.g., U_((fpp1)) is decreased as U_((ctc)) increases).

The Effect of Q(cbi) on UQ(fpp)

When a respondent responds incorrectly to a question after havingpreviously responded correctly to the question (thus incrementingQ_((cbi)) for the question), embodiments retroactively flag any previousresponses to the question as having a UQ_((fpp)) of 1. FIG. 20illustrates how a correct response may be flagged as a false positiveafter Question 9 is answered incorrectly. For example, User 1'sincorrect response of Question 9 (id 14) invalidates User 1's previouscorrect response of Question 9 (id 5). Accordingly, User 1's correctresponse to question 9 (id 5) is identified as a false positive. Inembodiments, flagging a response as a false positive triggers arecalculation of Q_((scav)), Q_((scsd)), Q_((siav)), and Q_((sisd)).

Particularly, in embodiments, whenever these values are recalculated, afalse positive causes the calculation to interchange the Null andnumeric values under Seconds to Correct (UQ_((sc))) and Seconds toIncorrect (UQ_((si))), respectively, prior to executing the calculation.Accordingly, response times for false positives are not counted in theQ_((scav)) and therefore, will not incorrectly skew the results of theUQ_((scsdfm)) or UQ_((fpp)) calculations of new responses to a question.

The Effect of User Tendencies on UQ_((fpp))

In embodiments, U_((tgf)) can be used as an operand in evaluating theUQ_((fpp)) for any correct response to a question. As discussed above,U_((tgf))=U_((gfc))/U_((tqa)). Regarding U_((gfc)), in embodiments,whenever a false positive with a probability of 1 is associated with auser's response (as in FIG. 20), U_((gfc)) is incremented accordingly.For example, using FIG. 20, U_(1(gfc)) would have a value of 1. Inembodiments, each new UQ_((fpp)) with a value of 1.00, regardless ofcontext (e.g. question was asked in Course A vs. Course B), incrementsU_((gfc)).

As an example, the higher a learner's tendency to correctly guess and/orforget a correct answer (U_((tgf)), the more likely that any givencorrect response by that learner is a false positive. In embodiments,the weight of this operand is mitigated by other operands, and has aminor overall effect on UQ_((fpp)).

In embodiments, the second operand in the UQ_((fpp)) calculation isUQ_((fpp2)). According to embodiments, UQ_((fpp2)) may be calculatedaccording to the following equation:

${UQ}_{({{fpp}\; 2})} = \frac{U_{{({tgf})}\;}}{{UQ}_{({ctc})}^{{UQ}_{({ctc})}}*4}$

According to the above equation, as illustrated in FIG. 21, UQ_((fpp2))increases as U_((tgf)) increases, but decreases as UQ_((ctc)) increases.UQ_((fpp2)) increases as U_((tgf)) increases since U_((tgf)) indicates auser's tendency to guess or forget a correct answer for each correctresponse the user provides. However, as discussed above, UQ_((fpp2))decreases as UQ_((ctc)) increases since a user is less likely to berandomly selecting the correct response if the user consistently selectsthe correct response.

FIG. 21 illustrates an extreme example where a user has been shown toguess or forget 99% of the time. When U_((tgf)) is 0.99, the falsepositive probability of that user's first consecutive correct responseto a question will be increased by 0.25. Further, the false positiveprobability of that user's second consecutive correct response to aquestion will only be increased by 0.06 (e.g., UQ_((fpp2)) decreases asU_((ctc)) increases). A more likely scenario is where a user has beenshown to guess or forget 25% of the time. When U_((tgf)) is 0.25, thefalse positive probability of that user's first consecutive correctresponse to a question will be increased by 0.06. Additionally, thefalse positive probability of that user's second consecutive correctresponse to a question will only be increased by 0.02.

The Effect of a Question's Quality of Distracters on UQ_((fpp))

The ability of incorrect option(s) to distract users from the correctoption(s) may be used to determine the likelihood that a user's correctresponse is a false positive. The more likely a user will select theincorrect response, the higher the ability of that incorrect response todistract the user from the correct response. FIG. 22 illustrates twoexample mathematical questions. As illustrated for Question 12, all theanswer options are numbers. However, for Question 13, option (d) is aword. Since option (d) is a word instead of a number, users are lesslikely to select option (d). Accordingly, option (d) is considered to bea low quality distracter.

In embodiments, the variable Q_((qod)) (Quality of Distracters) is ameasure of the quality of a question's distracters, measured on a scalebetween 0 (abysmal) and 1 (superb). In alternative embodiments, thescale may vary to any desired values to represent poor and high qualitydistracters. Poor quality distracters may be easier to eliminate fortheir lack of credibility, thereby increasing the likelihood ofselecting the correct answer without really knowing that the answer iscorrect. Thus, in embodiments, Q_((qod)) is factored into a question'sfalse positive probability Q_((fpp)).

In embodiments, the following is a non-exclusive list of variables usedto determine Q_((qod)):

Q_((tco)) Question Total Correct Options

-   -   For a set of options associated with a question, Q_((tco)) is        the total number of options that are correct.

Q_((tio)) Question Total Incorrect Options

-   -   For a set of options associated with a question, Q_((tio)) is        the total number of options that are incorrect.

Q_((ec)) Question Expected Correct

-   -   For a given question option, Q_((ec)) is the percentage of the        number of times that the correct options are expected to be        selected if all question options are equally believable to be        correct. In embodiments, Q_((ec))=Q_((tco))/Q_((to)), where        Q_((to)) represents the total number of answer options        associated with the question. As an example, if a question has 4        total options (Q_((to))) with 1 of those options designated as a        correct response (Q_((tco))), then Q_((ec))=¼=0.25, which means        that the correct response is expected to be selected 25% of the        time.

Q_((tos)) Question Total Options Selected

-   -   For a given question, Q_((tos)) is the sum of all QO_((ts)),        which represents the total number of times that an answer option        has been selected.

Q_((tcos)) Question Total Correct Options Selected

-   -   For a given question, Q_((tcos)) is the total number of correct        options selected.

Q_((tios)) Question Total Incorrect Options Selected

-   -   For a given question, Q_((tios)) is the total number of        incorrect options selected.

QO_((eps)) Question-Option Expected Percentage of Selections

-   -   For a given question option, QO_((eps)) is the percentage of the        number of times that the option is expected to be selected if        all question options are equally believable to be correct. In        embodiments, QO_((eps))=1/Q_((to)). As an example, if a question        has 4 answer options (Q_((to))), for the first option,        QO_(1(eps))=¼=0.25, which means that Option 1 is expected to be        selected 25% of the time.

QO_((b)) Question-Option Believability

-   -   For a given question option, this is the percentage of the time        that this option is actually selected. In embodiments,        QO_((b))=QO_((ts))/Q_((tos)). As an example, if a question with        three options has been presented 10 times (Q_((ts))), and the        first option is selected 3 times (QO_(1(ts))), QO_(1(b))=        3/10=0.3, which means that option 1 is selected 30% of the time.        QO_((dde)) Question-Option Distracter Deviation from Expected        Percentage of Selections    -   In embodiments, the following equation applies to distracters        only. If the answer option is correct, then QO_((dde))=0.

IF(QO _((eps)) −QO _((b))>0)THEN {QO _((dde)) =QO _((eps)) −QO _((b))}ELSE{QO _((dde))=0}

-   -   As illustrated in the preceding equation, if the percentage of        the number of times that an option is expected to be selected        (QO_((eps))) is greater than the percentage that the option is        actually selected (QO_((b))), QO_((dde)) is the difference        between QO_((eps)) and QO_((b)). Otherwise, QO_((dde)) is 0.        Q_((dde)) Question Distracter Deviation from Expected Percentage        of Selections    -   In embodiments, Q_((dde)) is the sum of all QO_((dde)) for a        given question.

In embodiments, Q_((qod)) may be calculated according to the followingequation:

${IF}\mspace{14mu} \left( {Q_{({sl})} > 0} \right)\mspace{14mu} {THEN}\left\{ {Q_{({qod})} = {{MAX}\left( {{{{MIN}\left( {{{MAX}\begin{pmatrix}{\left( {\left( \frac{Q_{({tios})}}{Q_{({tos})}*} \right)*\frac{1}{Q_{({tio})}}} \right)*} \\{\frac{1}{Q_{({ec})}*Q_{({tco})}},0}\end{pmatrix}},1} \right)} - Q_{({dde})}},0} \right)}} \right\} {ELSE}\mspace{14mu} \left\{ {Q_{({qod})} = 1} \right\}$

An example of a question type where Q_((s1))=0 is a fill in the blankquestion type. As illustrated in the above equation, Q_((qod)) mayrepresent the likelihood of selecting one or more incorrect options.

FIG. 23 illustrates a question (e.g., Question 1) with four answeroptions, one of which is correct, and three of which are incorrect. Ifeach answer option were equally believable to be correct, it is expectedthat each option is selected an equal number of times (QO_((eps)) is thesame for each option). As illustrated in FIG. 23, each answer option hadin fact been selected nine times, which also indicates that QO_((b)) isthe same for each question. This distribution of selections suggeststhat the distracters are of superb quality, as no distracter seems anyless believable than the correct answer option. Using the calculationabove for Q_((qod)), for this question, Q_((qod))=1.00. As anotherexample, FIG. 24 illustrates a Q_((qod)) value and values of operandsused in the Q_((qod)) equation for Question 4, where the distracters arenot selected an equal number of times.

FIG. 25A and FIG. 25B illustrate how the value of Q_((qod)) changes asthe distribution of answer option selections varies. For Question 2,only the correct answer option is selected. Thus, the quality of thedistracters for Question 2 is abysmal (e.g., none of the distractersseems to be a credible choice). Accordingly, Q_((qod)) is at theopposite (i.e. low) end of the scale (0.00) compared to Question 1(1.00). For Question 3, only the correct response option and onedistracter is selected, with the correct response selected more timesthan the distracter. Thus, one of the distracters seems somewhatcredible, but the other two distracters do not appear to be credible atall. Accordingly, Q_((qod)) is at the very low end of the scale (0.03)for Question 3.

For Question 4, only the correct response and two distracters areselected, with the correct response selected more times than the twodistracters. Thus, one of the distracters seems somewhat credible,another has minor credibility, and the other does not appear to becredible at all since it is never selected. Accordingly, Q_((qod)) isstill low (0.25) for Question 4.

For Question 5, the correct response and all the distracters areselected. As illustrated in FIG. 25A, there have been just as manyincorrect responses as correct responses. Therefore, the set ofdistracters as a whole distract users from the correct response.However, there are clear differences in the quality of one distracterversus another since one distracter is selected more than the otherdistracters combined. Accordingly, Q_((qod)) is medium-low (0.33) forQuestion 5.

For Question 6, the correct response and all three distracters areselected with the correct response selected more often than anyindividual distracter and each distracter selected an equal number oftimes. As illustrated in FIG. 25A, there have been just as manyincorrect responses as correct responses. Each distracter seems to be aseffective as the other two distracters since each distracter is selectedthe same number of times. However, no distracter is as credible as thecorrect answer since the correct answer is selected the most.Accordingly, Q_((qod)) is in the mid range (0.42) for Question 6.

For Question 7, the correct response and all three distracters areselected with one distracter selected the same number of times as thecorrect response. Q_((qod)) has surpassed the mid range (0.58). Asillustrated in FIG. 25B, there have been more incorrect responses thancorrect responses, which is what is expected when high qualitydistracters outnumber the correct answer options. However, onedistracter is clearly more credible than the other two distracters sincethat distracter is selected more often than the other two distracters.Accordingly, Q_((qod)) has surpassed the mid range (0.58) for Question7.

For Question 8, as illustrated in FIG. 25B, the incorrect responsesoutnumber the correct responses. As illustrated in FIG. 25B, respondentsare having a harder time discriminating among the distracters forQuestion 8 compared to Question 7. Accordingly, Q_((qod)) is medium-high(0.72) for Question 8.

For Question 9, as illustrated in FIG. 25B, Q_((qod)) is in the highrange (0.92) the incorrect responses again outnumber the correctresponses. As illustrated in FIG. 25B, since the number of selections ofdistracters for Question 9 is higher than Question 8, respondents areclearly having a harder time discriminating among the distracters forQuestion 9 compared to Question 8. Accordingly, Q_((qod)) is in the highrange (0.92) for Question 9.

For Question 10, the incorrect responses again outnumber the correctresponses. However, as illustrated in FIG. 25B, there is a clearpreference for one distracter over the other two distracters, whichindicates that the other two distracters are not as credible as thepreferred distracter. Accordingly, Q_((qod)) is still high for Question10, but has decreased (0.90) when compared to Question 9.

For Question 11, as illustrated in FIG. 17, the incorrect responsesoutnumber again outnumber the correct responses. However, compared toQuestion 10, there seems to be a common misconception regarding thedistracters for Question 11 since one of the distracters is selectedtwice as much as the other two distracters combined. Accordingly,Q_((qod)) for Question 11 has decreased (0.75) compared to Question 10.

FIG. 26 illustrates an example process for calculating Q_((qod)) for aparticular question. In embodiments, the process starts at 2600 where itis determined if Q_((s1)) greater than or equal to 0. If Q_((s1)) isgreater than 0, process flow proceeds from 2600 to 2602 where Q_((qod))may be calculated using the equation discussed above for Q_((qod)). Inembodiments, Q_((qod)) can be stored as a property of a particularquestion in the question database 308 (FIG. 3). Process flow proceedsfrom 2602 to 2606 to store Q_((qod)). If Q_((s1)) is less than or equalto 0 at 2600, process flow proceeds to 2604 where Q_((qod)) is set to 1.Process flow proceeds from 2604 to 2606 to store Q_((qod)). Inembodiments, the question module 408 performs the process illustrated inFIG. 26.

In embodiments, the variable Q_((fpo)) (Question False Positive Offset)is a value that strengthens the effect of Q_((prc)) on UQ_((fpp)) as thevalue of Q_((qod)) decreases. As an example, when a prudent test takerdoes not know the correct answer to a question, his first strategy canbe not to select an answer at random. Instead the test taker attempts toreduce the number of credible answer options by process of elimination.If the test taker can reduce the number of credible answer options fromfive down to three, for example, he will have better odds of guessingthe correct answer, thus increasing the odds of registering a falsepositive. Distracters of high quality are more difficult to eliminatethan low quality distracters. Particularly, as the quality ofdistracters increase, the percent chance of randomly selecting thecorrect answer from a set of answer options decreases. The followingequation illustrates how Q_((qod)) mitigates Q_((prc)) for the purposesof estimating UQ_((fpp)):

IF  (Q_((qod)) = 0)  THEN  {Q_((qod)) = 0.01}$Q_{({fpo})} = {{MIN}\left( {{{MAX}\left( {\frac{Q_{({prc})}}{Q_{({qod})}},0.01,} \right)},0.99} \right)}$

In embodiments, Q_((fpo)) is not used in the Certainty mode.

FIG. 27 illustrates how Q_((fpo)) varies when Q_((prc)) and Q_((qod))varies. As illustrated in FIG. 27, when the quality of distracters ishigh (i.e., when Q_((qod))=1.00), then the percent random chance ofselecting the correct answer Q_((prc)) is not affected by Q_((qod)).That is, since all distracters are treated as equally credible,Q_((fpo))=Q_((prc)). However, as the quality of the distractersdegrades, the percent random chance of selecting the correct answerincreases (e.g., it is easier for the user to disqualify the poordistracter and randomly select the correct choice). Therefore, a lowerQ_((qod)) effectively raises Q_((prc)) as illustrated in FIG. 27.Accordingly, Q_((fpo)) represents the probability of guessing the rightanswer to a question after taking into account the quality of thedistracters Q_((qod)) of the question.

In embodiments if Q_((qod))=0, then every response submitted for thatquestion has been a correct response. This situation does not alwaysindicate a definite false positive where UQ_((fpp)) would equal 1.00.This situation may indicate that the question itself is too easy for theaudience, or that the distracters are absurd alternatives to the correctanswer. In either case, the frequency that this question is re-presentedto the learner, in embodiments, will be dramatically reduced by theUQ_((ctc)) factor (e.g., as UQ_((ctc)) increases, the likelihood ofseeing the question again decreases). However, the variable UQ_((fpp))is increased by Q_((fpo)) since the latter case represents a situationin which the learner is more likely to get the answer right withoutnecessarily knowing the answer.

In embodiments, the third operand in the UQ_((fpp)) calculation isUQ_((fpp3)) calculated by the following equation:

${UQ}_{({{fpp}\; 3})} = \frac{Q_{({fpo})}}{{UQ}_{{({ctc})}\;}^{{UQ}_{({ctc})}}}$

As discussed above, Q_((fpo)) is correlated with the percent randomchance of selecting the correct answer Q_((prc)) after taking intoaccount the quality of the distracters Q_((qod)). As illustrated in theabove equation and FIG. 28, UQ_((fpp3)) increases as Q_((fpo))increases. If Q_((fpo))=0.99, then there is a 99% chance of correctlyguessing the correct answer. In this extreme example , if the user hassubmitted only one consecutive correct response, UQ_((fpp3))=0.99.However, if the user has submitted two consecutive correct responses,UQ_((fpp3))=0.25. In a more likely scenario where Q_((fpo))=0.25 (e.g.,there is a 25% chance of correctly guessing the correct answer, ascalculated from a Q_((qod)) of 0.80 and a Q_((prc)) of 0.20) if the userhas submitted only one consecutive correct response, UQ_((fpp3))=0.25.If the user has submitted two consecutive correct responses,UQ_((fpp3))=0.06.

Calculating UQ_((fpp3))=0.06.

Generally, a method for submitting a response to a question includesclicking a “Submit” button or some variation of the same type ofsubmission method. In embodiments, “Certainty Mode” replaces the“Submit” button with a mechanism for reporting the respondent'sassuredness that his response is correct. FIG. 29 illustrates how anembodiment of the Certainty Mode feature replaces a standard Submitbutton. The left side of FIG. 29 illustrates a first question 2900 withanswer options 2902-2906. The question 2900 is in standard submissionmode where a user clicks the “Submit” button 2908 to submit a response.The right side of FIG. 29 illustrates the question 2910 a second timewith answer options 2912-2916. However, the second instance of thequestion 2910 is in the “Certainty Mode” where the user is presentedwith a scale 2918 and submit button 2920 that slides along the scale2918, which permits a user to report a level of confidence for aresponse. For example, if the user is highly confident in a response,the user may slide the submit button 2920 to the “Reasonably Sure” sideof the scale 2918 and click the submit 2920 button. If the user is notconfidant in the selected response, the user may slide the submit button2918 to the “Mostly Guessing” side of the scale and click the submitbutton. In embodiments, the actual wording (e.g. “Mostly Guessing”) andthe format of the scale may vary to any desired wording and format,respectively. Embodiments may use a sliding scale of 0% to 100%, wherethe higher the percentage reported, the more confident the user is inthe selected response.

FIG. 30 illustrates an example process for displaying a question in theCertainty Mode feature. Process flow begins at 3000 where a question isdisplayed in the “Certainty Mode.” In embodiments, process flow proceedsfrom 3000 to 3002 where a scale with a button displaying “drag” isdisplayed to the user. If the user clicks the button without indicatinga level of confidence, process flow proceeds from 3002 to 3004 where theuser is notified that the user must indicate a degree of certainty priorto submitting a response to the question. In embodiments, thisnotification can be displayed in a pop-up window where the user canacknowledge receipt of the notification by pressing an “ok” button. Uponacknowledging receipt of the notification, process flow returns from3004 to 3002. Upon moving a mouse cursor over the button, process flowproceeds from 3002 to 3006 where the user receives instructions to movethe button to the left or right side of the scale.

Process flow proceeds from 3006 to 3008 if the user moves the button tothe left or right side of the scale. Process also proceeds from 3002 to3008 if the user moves the button to the left or right side of thescale. Upon placing the button on the left or right side of the scale,process flow proceeds from 3008 to 3010 where the button display ischanged from “drag” to “submit.” Upon placing a cursor over the button,process flow proceeds from 3010 to 3012 where the user receivesinstructions to move the button left or right or click the button tosubmit a selected response to a question. If the user further moves thebutton, process flow returns from 3012 to 3008 where the steps describedabove for 3008 are repeated.

If the user clicks the button, process flow proceeds from 3012 to 3014where the user receives notification that the user's response issubmitted. Process flow proceeds from 3014 to 3016 to disable the dragdrop functionality of the button. Additionally, in embodiments, when theuser clicks the button, a % certainty can be determined using a distancefrom the leftmost point of the scale (0) and toward the rightmost pointof the scale (100). The user's response and % certainty are submitted.

In embodiments, the variable S_((ct)) (System Certainty Threshold)represents a degree of certainty (e.g. 85%) below which any correctresponse by any user is treated as a false positive. In embodiments,U_((ct)) (User Certainty Threshold) represents a degree of certainty(e.g., 85%) below which any correct response by a particular user istreated as a false positive. For example, embodiments may use a seriesof test questions to evaluate the correlation between an individual'swillingness to act in the face of the individual's reported level ofcertainty. Particularly, the series of test questions may reveal that auser is a natural risk taker since the user is more likely to select aresponse even though the user is not certain that the response iscorrect. Alternatively, the series of test questions may reveal that theuser is conservative since the user is not likely to select a responseunless the user is certain that the response is correct. Accordingly,users identified as natural risk takers may have U_((ct))=40% whereasusers identified as conservative may have U_((ct))=75% since theconservative user is not likely to select a response unless theconservative user is certain that the response is correct. Inembodiments, if no U_((ct)) is specified, U_((ct))=S_((ct)).

In embodiments, UQ_((cw)) (Certainty Weight) is a value that usesUQ_((ctc)), UQ_((c)), and U_((ct)) or S_((ct)) using the following threepoint gradient calculation:

$\begin{matrix}{{{IF}\mspace{14mu} \left( {{{{{UQ}_{({ctc})} > 0}\&}{UQ}_{(c)}}>=U_{({ct})}} \right)}\left\{ {{{UQ}_{({c\; w\; 1})} = {- \left( \begin{matrix}{\left( {{UQ}_{(c)} - U_{({ct})}} \right)*} \\{{UQ}_{({ctc})}*2^{{UQ}_{({ctc})}}}\end{matrix} \right\}}};{{UQ}_{({cw})} = {UQ}_{({{cw}\; 1})}}} \right\} {{ELSE}\mspace{14mu} {IF}\mspace{14mu} \left( {{{{{UQ}_{({ctc})} > 0}\&}{UQ}_{(c)}} < U_{({ct})}} \right)}\left\{ {{{UQ}_{({{cw}\; 2})} = {- \left( \frac{\left( {{UQ}_{(c)} - U_{({ct})}} \right)*2^{{UQ}_{({ct})}}}{10} \right)}};{{UQ}_{({cw})} = {UQ}_{({{cw}\; 2})}}} \right\} {{ELSE}\mspace{14mu} {IF}\mspace{14mu} \left( {{{{{UQ}_{({ctc})} < 0}\&}{UQ}_{(c)}}>=U_{({ct})}} \right)}\left\{ {{{UQ}_{({{cw}\; 3})} = \frac{\begin{pmatrix}{\left( {\begin{pmatrix}{\left( {{UQ}_{(c)}*10} \right) +} \\2^{{{UQ}_{({ctc})}}*{{UQ}_{({ctc})}}}\end{pmatrix}*2} \right) +} \\{{{U_{({ct})} - {UQ}_{(c)}}}*100}\end{pmatrix}}{10}};{{UQ}_{({({cw})}} = {UQ}_{({{cw}\; 3})}}} \right\}} & \; \\{{{ELSE}\mspace{14mu} {IF}\mspace{14mu} \left( {{{{{UQ}_{({ctc})} < 0}\&}{UQ}_{(c)}} < U_{({ct})}} \right)}\left\{ {{{UQ}_{({{cw}\; 4})} = \frac{\left( {{UQ}_{(c)}*10} \right) + 2^{{{UQ}_{({ctc})}}*{{UQ}_{({ctc})}}}}{10}};{{UQ}_{({cw})} = {UQ}_{({{cw}\; 4})}}} \right\}} & \;\end{matrix}$

FIG. 31 illustrates how UQ_((cw)) varies when the Certainty Threshold is80% (U_((ct))). FIG. 32 illustrates how UQ_((cw)) varies when theCertainty Threshold is 50% (U_((ct))). The columns of FIG. 31 and FIG.32 indicate a user's reported level of confidence (U_((c))). The rows ofFIG. 31 and FIG. 32 indicate a particular UQ_((ctc)) value. Further, thecells in FIG. 31 and FIG. 32 indicate a particular UQ_((cw)) calculatedusing the equation above depending on the values of U_((c)), UQ_((ctc)),and U_((ct)). As discussed above, the Certainty Threshold, whether apredetermined system-wide setting or an individually determined value,is an estimate of the degree of certainty at which an individual islikely to act on information that the individual believes to be correct.

In embodiments, a positive UQ_((cw)) will increase UQ_((sw)) (stackingweight), and a negative UQ_((cw)) will decrease UQ_((sw)). Accordingly,in embodiments, the larger the UQ_((cw)) value is, the more likely it isthat an ILAT running in Certainty Mode will present the question to theuser again in the near future. The smaller the UQ_((cw)) value is, theless likely it is that the ILAT will present the question to the useragain in the near future. For example, referring to FIG. 31, a user whoresponds correctly to a question for the second consecutive time andreports an 85% certainty level has UQ_((sw)) decrease by 0.40, thusdecreasing the likelihood that the user will see this question again inthe near future. However, had the user answered incorrectly for thefirst time at a certainty level of 85% UQ_((sw)) would increase by 2.60,thus increasing the likelihood that the user will see this questionagain in the near future.

As illustrated in FIG. 31 and FIG. 32, if the user consecutively answersa question correctly and reports a high level of confidence (upper rightquadrant), UQ(cw) decreases in value, which decreases the likelihoodthat the user will see the question again. If the user consecutivelyanswers the question incorrectly and reports a level of confidence lowerthan U_((ct)), UQ_((cw)) increases in value by a relatively smalldegree, which somewhat increases the likelihood that the user will seethe question again. An incorrect response with UQ_((c)) higher thanU_((ct)) increases UQ_((cw)) by a more substantial degree, whichsignificantly increases the likelihood that the user will see thequestion again. As further illustrated in FIG. 31 and FIG. 32, if a useranswers a question incorrectly five times in a row, the user is morelikely to see the question again if the user reports a high level ofcertainty compared to if the user reports a low level of certainty.Accordingly, a goal of the system is that the users' responses tendtoward the upper right quadrant of FIG. 31 and FIG. 32 and away from thelower left quadrants.

FIG. 33 illustrates an example process of calculating UQ_((cw)). Theprocess starts at 3300 to determine if the system is in Certainty Mode.If the system is not in Certainty Mode, process flow proceeds from 3300to 3302 to set UQ_((cw)) to zero. That is, when not in Certainty Mode,UQ_((cw)) has no affect on UQ_((sw)). If the system is in CertaintyMode, process flow proceeds from 3300 to 3304 to determine if UQ_((ctc))is above or below zero. If UQ_((ctc)) is above zero, process flowproceeds from 3306 to determine if UQ_((c)) is greater than or equal toU_((ct)). If UQ_((c)) is greater than or equal to U_((ct)) (i.e., theuser's reported confidence level is greater than or equal to theCertainty Threshold), process flow proceeds from 3306 to 3308 tocalculate UQ_((cw1)) as indicated above. If UQ_((c)) is less thanUQ_((ctc)) (i.e., the user's reported confidence level is less than theCertainty Threshold), process flow proceeds from 3306 to 3310 tocalculate UQ_((cw2)) as indicated above.

If UQ_((ctc)) is less than zero, process flow proceeds from 3304 to 3312to determine if UQ_((c)) is greater than or equal to U_((ct)). IfUQ_((c)) is greater than or equal to U_((ct)), process flow proceedsfrom 3312 to 3316 to calculate UQ_((cw3)) as indicated above. IfUQ_((c)) is less than U_((ct)), process flow proceeds from 3312 to 3314to calculate UQ_((cw4)) as indicated above. Process flow proceeds from3308, 3310, 3314, or 3316 to 3318 to store UQ_((cw)). In embodiments,UQ_((cw)) can be stored in the user history response database 310 (FIG.3). In embodiments, the evaluation module (410) performs the processillustrated in FIG. 33.

Calculating UQ(fpp)

As discussed above, UQ_((fpp)) (User-Question False PositiveProbability, is an estimate of the likelihood that a response to aquestion is a false positive. UQ_((fpp)) may be a value with two decimalplaces ranging between 0.00 and 1.00. In embodiments, when not runningin Certainty Mode, UQ_((fpp)) equals:

UQ _((ƒpp))=MIN(MAX(UQ _((ƒpp1)) +UQ _((ƒpp2)) +UQ _((ƒpp3)),0),0.99)

FIG. 34 shows an example process for determining UQ_((fpp)). The processstarts at 3400 to determine if UQ_((ctc))<0. If UQ_((ctc))<0, processflow proceeds from 3400 to 3402 to set UQ_((fpp)) to 0. As an example,if UQ_((ctc))<0, the current response to the question is incorrect,which means that the response cannot be a false positive. Process flowproceeds from 3402 to 3414 to store UQ_((fpp)).

If UQ_((ctc))>0, process flow proceeds from 3400 to 3404 to determine ifthe system is running in Certainty Mode 3404. If the system is runningin Certainty Mode, process flow proceeds from 3404 to 3406 to determineif UQ_((c))<U_((ct)). Accordingly, if the user's reported level ofconfidence is less than the Certainty Threshold, process flow proceedsfrom 3406 to 3408 where the user's response is treated as a definitefalse positive and UQ_((fpp))=1. If the user's reported level ofconfidence is greater than or equal to the Certainty Threshold, processflow proceeds from 3406 to 3412 to determine UQ_((fpp)) usingUQ_((fpp1)), UQ_((fpp2)), and UQ_((fpp3)) as indicated above. Processflow proceeds from 3408 and 3410 to 3414 to store UQ_((fpp)).

If the system is not running in Certainty Mode, process flow proceedsfrom 3404 to 3412 to determine UQ_((fpp)) using UQ_((fpp1)),UQ_((fpp2)), and UQ_((fpp3)) as indicated above. Process flow proceedsfrom 3412 to 3414 to store UQ_((fpp)). UQ_((fpp)) may be stored in theuser history response database 310 (FIG. 3). In embodiments, theevaluation module 410 (FIG. 4) performs the process illustrated in FIG.34.

Calculating UQ(sw)

As discussed above, UQ_((sw)) is a stacking weight associated with aquestion where an increase in UQ_((sw)) increases the likelihood thatthe question will be re-presented to a user. In embodiments, UQ(sw) maybe determined from the following equation:

${IF}\mspace{14mu} \left( {{UQ}_{({ctc})} > 0} \right)\begin{Bmatrix}\begin{matrix}{\left. {{UQ}_{({{sw}\; 1})} = {\left( {UQ}_{({ctc})}^{Q_{({ctc})}} \right)*\left( {- 1} \right)}} \right) +} \\{\left. {{UQ}_{({fpp})} + {UQ}_{({cw})}} \right);}\end{matrix} \\{{UQ}_{({sw})} = {UQ}_{({{sw}\; 1})}}\end{Bmatrix}$ELSE  IF  (UQ_((ctc)) < 0){UQ_((sw 2)) = UQ_((ctc))^(UQ_((ctc))); UQ_((sw)) = UQ_((sw 2))}

FIG. 35 illustrates an example process for determining UQ_((sw)). Theprocess starts at 3500 to determine if UQ_((ctc))>0. If UQ_((ctc))is >0, process flow proceeds from 3500 to 3502 to determine UQ_((sw1))as indicated above. Process flow proceeds from 3502 to 3506 to storeUQ_((sw1)) as UQ_((sw)). If UQ_((ctc)) is <0, process flow proceeds from3500 to 3504 to determine UQ_((sw2)) according to the equation above.Process flow proceeds from 3504 to 3506 to store UQ_((sw2)) asUQ_((sw)). Process flow proceeds from 3506 to 3508 to return UQ_((sw))to the ILAT. UQ_((sw)) may be stored in the user's history responsedatabase 310 (FIG. 3). In embodiments, the evaluation module 410 (FIG.4) performs the process illustrated in FIG. 35.

FIG. 36 illustrates an example process calculating a recommendedstacking weight and its constituent operands. The steps highlighted inblack illustrates steps described in FIG. 1. As discussed above, anymechanism or device that determines one or more operands to determineUQ_((fpp)) and/or UQ_((sw)) may be external to an ILAT or,alternatively, may be incorporated as part of the ILAT. In embodiments,the process illustrated in FIG. 36 uses and updates the variablesUX_((crsr)) (User Course-Lesson-Module) Correct Responses SinceRemediation) and UX_((irsr)) (User Course-Lesson-Module IncorrectResponses Since Remediation). UX_((crsr)) represents the total number ofcorrect responses that a user has made since the last instructionalinterval was reached. UX_((irsr)) represents a total number of incorrectresponses that a user has made since the last instructional interval wasreached. X refers to the container of the question such as Course and/orLesson and/or Module and/or any other desired identifier.

The process begins at 3600 where at least a question stack, aninstruction interval, and a performance threshold are assigned asdescribed above. Process flow proceeds from 3600 to 3602 where the userreceives a question stack as previously described. Process flow proceedsfrom 3602 to 3604 to determine if the instruction interval reached isreached as previously described. If the instruction interval is reached,process flow proceeds from 3604 to 3606 to instruct the user aspreviously described. Process flow proceeds to 3608 to set UX_((crsr))and UX_((irsr)) to 0 since the instruction interval is reached. Processflow proceeds to 3610 to determine if the user's performance thresholdis reached as previously described. If the instruction interval is notreached, process flow also proceeds from 3604 to 3610. If the user'sperformance threshold is reached, the process ends 3612.

If the user's performance threshold is not reached, process flowproceeds from 3610 to 3614 where the next question is selected fordisplay as previously described. Process flow proceeds to 3616 to recordthe current time as the display time (UQ_((dt))) of the selectedquestion. Process flow proceeds to 3618 to determine if the selectedquestion should be displayed in Certainty Mode as previously described.Upon determining whether the selected question should be displayed inCertainty Mode, process flow proceeds to 3620 to display the selectedquestion and determine if a response is received for the selectedquestion 3622 as previously described.

If no response is received for the selected for the displayed question(e.g., as determined by the expiration of a time limit, or the userexiting the course, etc.), the process ends. If a response is receivedfor the selected question, process flow proceeds from 3622 to 3624 wherethe user's response to the selected question is submitted, and the timethat the response is received is recorded as UQ_((rt)). Process flowproceeds to 3624 to increment the total number of questions answered bythe user (U_((tqa))). Additionally UX_((crsr)) or UX_((irsr)) dependingon whether the received response is correct or incorrect, respectively.Process flow proceeds to increment QO_((ts)) to indicate the number oftimes the user's current response is selected.

Process flow proceeds to 3630 to determine if the user's currentresponse is correct. If the user's current response is not correct,process flow proceeds from 3630 to 3632 to increment the total number oftimes the question has been answered incorrectly (Q_((tti))), and recordthe time that it took to answer the question incorrectly (UQ_((si))). Ifthe user's current response is correct, process flow proceeds from 3630to 3634 to increment the total number of times the question has beenanswered correctly (Q_((ttc))), and record the time that it took toanswer the question correctly (UQ_((sc))). Process flow proceeds from3632 or 3634 to 3636 to update the variables UQ_((sct)), UQ_((sc)),Q_((scav)), Q_((scsd)), and UQ_((scse)) as described above to determinehow quickly the user responded to the question compared to other users.

Process flow proceeds from 3636 to 3638 to determine UQ_((scsdfm)) asdescribed above. Process flow proceeds to 3640 to calculate one or moreoperands, described below, used to determine the likelihood that theuser's correct response to the selected question is a false positive(among other operands described above), and determine the recommendedstacking weight of the question. Process flow proceeds to 3642 to returncalculated values to the ILAT.

FIG. 37 illustrates one or more operands determined by the processillustrated in FIG. 36 at 3638. The operands include, but are notlimited to, UQ_((ctc)) 3700, Q_((qod)) 3702, Q_((cbi)) 3704, Q_((pir))3706, UQ_((fpp)) 3708, UQ_((cw)) 3710, and UQ_((sw)) 3712.

Support for Diverse Types of Questions

In embodiments, to support problem types other than the selection of anoption or option set from a larger option set (as in true/false ormultiple-choice questions), options of a question may be any desiredresponse type. For example, suppose a question is “The Spanish word for‘cat’ is ______.” In this case, the respondent is required to type theanswer to the question, as opposed to selecting the correct answer froma list.

FIG. 38 illustrates an example properties list for Q₇, which is a fillin the blank question type. Values for Q_((to)) and Q_((s1)) are set to0. In embodiments, Q_((prc)) is estimated and manually input by theauthor or editor of the question as a value that is in a range not lessthan 0 and not greater than 50 (i.e. not greater than the Q_((prc)) of aTrue/False question).

In embodiments, an answer option value for Q₇ could be a single word, oran array, or any desired value representative of the correct response tothe question. For example, if Q₇ is a software simulation, Q₇O₁ may beused to store a target frame number of a movie, or “correct state”variable, or a set of acceptable paths to arrive at an end state of abranching exercise, etc. That is, the programming logic of aninteractive question type is free to determine whether a response meetsthe criteria for a particular option to have been “selected” and whetherto direct the system to increment the Q_((ts)), Q_((ttc)), and Q_((tti))values for that question. In embodiments, however, values for Q_((cbi))and Q_((pir)) are updated by the invention whenever Q_((ttc)) orQ_((tti)) are updated.

In embodiments, an answer option to a question is any desired mechanismsuitable for receiving an appropriate response to the question based onthe type of question.

For example, if the question is a fill-in-the blank question type, ananswer option for this question can be a single word. As anotherexample, if the question is a short answer question type, an answeroption for this question can be a paragraph consisting of one or moresentences.

Maintaining Granular Data on Questions, Options, Responses andRespondents

In embodiments, a record of all a respondent's responses to a questionis maintained. FIG. 39 illustrates example data of User 1's response toQuestion 3. Options selected by User 1 are designated by 1s in theResponse column, whereas options not selected are designated by 0s. Aselection (or lack of selection) that is correct may be designated by a1 in the Response Correctness column. A selection (or lack of selection)which is incorrect may be designated by a 0 in the Response Correctnesscolumn. As illustrated in FIG. 39, User 1 selected O₂ for Question 3,but did not select O₁, O₃, and O₄. As further illustrated in FIG. 39,the non-selection of O₃ and O₄ is correct, while the non-selection of O₁and selection of O₂ is incorrect. In embodiments, the ResponseCorrectness is not stored since this value may be calculated in realtime.

Suppose that the ILAT presents Q₃ for a second time to User 1. Thistime, as illustrated in FIG. 40, User 1 incorrectly selects O₃. Inembodiments, values in the id column are auto-incremented so that anorder of responses can be compiled. For example, as illustrated in FIG.40, the second response to O₃ is recorded, but the order of thoseresponses can be determined by their auto-incremented id (i.e. id₍₃₎precedes id₍₇₎). That is, the first time Question 3 was presented toUser 1, id₍₃₎ in FIG. 39 indicates that User 1 did not select O₃.However, the second time Question 3 was presented to User 1, id₍₇₎ ofFIG. 40 indicates that User 1 selected O₃. Any other desired orderingmechanism (e.g. a timestamp) can be used in place of the “id” column.

In embodiments, multiple responses of multiple respondents for multiplequestions of diverse question types are stored as illustrated in FIG.41. As illustrated in FIG. 41, the response column stores inputtedresponses to a fill-in-the-blank type question.

Control Panel

In embodiments, a control panel is provided to an administrator of anILAT. FIGS. 42A-B illustrate an example control panel. As illustrated inFIGS. A-B, the control panel permits the administrator to select whichoperands the system will return back to the ILAT. The control panel alsopermits the administrator to enable or disable the Certainty Mode, andto set a Default Certainty Threshold S_((ct)). In embodiments, the tunermodule 414 (FIG. 4) implements the control panel illustrated in FIGS.42A-B. The following is a non-exclusive list of variables listed in thecontrol panel (FIGS. 42A-B):

Q_((cbi)) Question: Correct Before Incorrect

-   -   The sum of all UQ_((cbi)) for a given question.        Q_((dde)) Question: Distracter Deviation from Expected        Percentage of Selections    -   The sum of all QO_((dde)) for a given question.

Q_((ec)) Question: Expected Correct

-   -   For a given question option, Q_((ec)) is the percentage of the        number of times that the correct options are expected to be        selected if all question options are equally believable to be        correct. Q_((ec))=Q_((tco))/Q_((to))

Q_((pir)) Question: Percentage of Incorrect Responses

-   -   Of the total number of respondents' responses to a problem,        Q_((pir)) is the percentage of those responses that are        incorrect or considered false positives, expressed as a decimal.

Q_((prc)) Question: Percent Random Chance

-   -   The percent random chance of selecting a correct response to a        question.

Q_((qod)) Question: Quality of Distracters

-   -   A measure of the quality of a question's distracters, measured        on a scale between 0 (abysmal) and 1 (superb).

Q_((sc)) Question: Seconds to Correct

-   -   A set of values consisting of all UQ_((sc)) for a given        question.

Q_((scav)) Question: Seconds to Correct Average

-   -   The average value of Q_((sc)).

Q_((scsd)) Question: Seconds to Correct Standard Deviation

-   -   The standard deviation of Q_((sc)).

Q_((si)) Question: Seconds to Incorrect

-   -   A set of values consisting of all UQ_((si)) for a given        question.

Q_((siav)) Question: Seconds to Incorrect Average

-   -   The average value of Q_((si)).

Q_((sisd)) Question: Seconds to Incorrect Standard Deviation

-   -   The standard deviation of Q_((si)).

Q_((s1)) Question: Selection Limit

-   -   Represents a limit on the number of answer options that a user        is allowed to or instructed to select.

Q_((tco)) Question: Total Correct Options

-   -   For a set of options associated with a question, Q_((tco)) is        the total number of options that are correct.

Q_((tcos)) Question: Total Correct Options Selected

-   -   For a given question, Q_((tcos)) is the total number of correct        options selected.

Q_((tio)) Question: Total Incorrect Options

-   -   For a set of options associated with a question, Q_((tio)) is        the total number of options that are incorrect.

Q_((tios)) Question: Total Incorrect Options Selected

-   -   For a given question, Q_((tios)) is the total number of        incorrect options selected.

Q_((to)) Question: Total Options

-   -   Represents the total number of answer options associated with a        question.    -   This is an important value for calculating Q_((prc)).

Q_((tos)) Question: Total Options Selected

-   -   For a given question, Q_((tos)) is the sum of all QO_((ts)).

Q_((ttc)) Question: Total Times Correct

-   -   The total number of times that a correct response has been        submitted for a question.

Q_((tti)) Question: Total Times Incorrect

-   -   The total number of times that an incorrect response has been        submitted for a question.

QO_((b)) Question-Option: Believability

-   -   For a given question option, QO_((b)) is the percentage of the        time that this option is actually selected.        QO_((b))=QO_((ts))/Q_((tos))

QO_((c)) Question-Option: Correct

-   -   This value designates whether an answer option is correct or        incorrect.        QO_((dde)) Question-Option: Distracter Deviation from Expected        Percentage of Selections    -   In embodiments, the following equation applies to distracters        only. If the answer option is correct, then QO_((dde))=0. IF        (QO_((eps))−QO_((b))>0) THEN {QO_((dde))=QO_((eps))−QO_((b))}        ELSE {QO_((dde))=0}

QO_((eps)) Question-Option: Expected Percentage of Selections

-   -   For a given question option, QO_((eps)) is the percentage of the        number of times that the option is expected to be selected if        all question options are equally believable to be correct.        QO_((eps))=1/Q_((to)).

QO_((ts)) Question-Option: Times Selected

-   -   Designates the total number of times that an answer option has        been selected.

S_((ct)) System: Certainty Threshold

-   -   When running in Certainty Mode, S_((ct)) is a value for a degree        of certainty below which any correct response by any user is        treated as a false positive.

U_((ct)) User: Certainty Threshold

-   -   When running in Certainty Mode, U_((ct)) is a value for a degree        of certainty below which any correct response by a particular        user is treated as a false positive. If no U_((ct)) is        specified, U_((ct))=S_((ct)).

U_((gfc)) User: Guess/Forget Count

-   -   The total number of definite false positives logged by the        system for a particular user.

U_((tgf)) User: Tendency to Guess or Forget

-   -   U_((tgf))=U_((gfc))/U_((tqa)).        U_((tqa)) User: Total Questions asked    -   The total number of questions that have been displayed to a        particular user.

UQ_((c)) User-Question: Certainty

-   -   A user's reported level of certainty that a submitted response        is correct.

UQ_((cr)) User-Question: Correctness

-   -   Indicates whether a user's response to a question was correct or        incorrect.

UQ_((cbi)) User-Question: Correct Before Incorrect

-   -   A count of the number of times that a respondent correctly        responds to a problem prior to incorrectly responding to that        problem.

UQ_((ctc)) User-Question: Consecutive Times Correct

-   -   The number of times that a user's chronological history of        responses to a question terminates with one or more consecutive        correct responses (in which case UQ_((ctc)) is expressed as a        positive integer), or one or more consecutive incorrect        responses (in which case UQ_((ctc)) is expressed as a negative        integer).

UQ_((cw)) User-Question: Certainty Weight

-   -   When run in Certainty Mode, given UQ_((c)), U_((ct)), and        UQ_((ctc)), UQ_((cw)) is a calculated value that raises or        lowers UQ_((fpp)) and UQ_((sw)).

UQ_((dt)) User-Question: Display Time

-   -   A timestamp of when an instance of a question was displayed to a        user.

UQ_((fpp)) User-Question: False Positive Probability

-   -   Estimates the probability that a response to a question is a        false positive.

UQ_((rca)) User-Question: Response Correctness Array

-   -   A chronological history of correctness of a learner's set of        responses to a question.

UQ_((rt)) User-Question: Response Time

-   -   A timestamp of when an instance of a question was responded to        by a user.

UQ_((sc)) User-Question: Seconds to Correct

-   -   An amount of time that has passed between a question being        displayed to a user and the user answering the question        correctly.        UQ_((scsdfm)) User-Question: Seconds to Correct, Standard        Deviations from the Mean=CEILING(ABS(MIN(UQ_((scse)),0)),1)

UQ_((scse)) User-Question: Seconds to Correct, Normalized Value

-   -   The normalized value of a UQ_((sc)) given Q_((sc)) for that        question where

$Z = {\frac{X - \mu}{\sigma}.}$

UQ_((si)) User-Question: Seconds to Incorrect

-   -   An amount of time that has passed between a question being        displayed to a user and the user answering the question        incorrectly.

UQ_((sw)) User-Question: Recommended Stacking Weight

-   -   A value that combines UQ_((ctc)), UQ_((sw)), and UQ_((fpp)).

UQ_((ttc)) User-Question: Total Times Correct

-   -   The total number of times that a specific user has submitted a        correct response for a specific question.

UQ_((tti)) User-Question: Total Times Incorrect

-   -   The total number of times that a specific user has submitted an        incorrect response for a specific question.

UQO_((r)) User-Question-Option: Response

-   -   A user's response to a question option.

UQO_((rc)) User-Question-Option: Response Correctness

-   -   The correctness of a user's response to a question option.

UX_((crsr)) User (Course-Lesson-Module) Correct Responses SinceRemediation

-   -   The total number of responses that a user has made since the        last instructional interval was reached.

UX_((irsr)) User (Course-Lesson-Module) Incorrect Responses SinceRemediation

-   -   The total number of responses that a user has made since the        last instructional interval was reached.

Example Computing Device

Embodiments of the present invention include various steps, which aredescribed herein. The steps may be performed by hardware components ormay be embodied in machine-executable instructions, which may be used tocause a general-purpose or special-purpose processor programmed with theinstructions to perform the steps. Alternatively, the steps may beperformed by a combination of hardware, software, firmware or humanrepresentatives of the parties or entities involved in the transaction.

FIG. 43 is a schematic diagram of a computing device 4300 upon which thelearning center 200 (FIG. 2) and/or other aspects of FIG. 2 may execute.In embodiments, the computing device is, for example, configured toperform the process illustrated in FIG. 36.

In embodiments, the computing device 4300 includes a bus 4301, at leastone processor 4302, at least one communication port 4303, a main memory4304, a removable storage media 4305 a read only memory 4306, and a massstorage 4307. Processor(s) 4302 can be any know processor, such as, butnot limited to, an Intel® Itanium® or Itanium 2® processor(s), or AMD®Opteron® or Athlon MP® processor(s), or Motorola® lines of processors.

Communication port(s) 4303 can be any known communications conduit suchas, but not limited to, an RS-232 port for use with a modem based dialupconnection, a 10/100 Ethernet port, or a Gigabit port using copper orfiber. In embodiments, the processor 4302 can be configured to executethe modules illustrated in FIG. 4.

Communication port(s) 4303 may be chosen depending on a network such asa Local Area Network (LAN), Wide Area Network (WAN), or any desirednetwork to which the computing device 4300 connects. The computingdevice 4300 may be in communication with peripheral devices such as, butnot limited to, printers, speakers, cameras, microphones, or scanners.

Main memory 4304 can be Random Access Memory (RAM), or any other desiredstorage device(s). Read only memory 4306 (ROM) can be any suitablestorage device(s) such as, but not limited to, Programmable Read OnlyMemory (PROM) chips for storing static information such as instructionsfor processor 4302. In embodiments, the ROM 4306 can be configured tostore any information in the databases illustrated in FIG. 3.

Mass storage 4307 can be used to store information and instructions. Forexample, hard disks such as the Adaptec® family of SCSI drives, anoptical disc, an array of disks such as RAID, such as the Adaptec familyof RAID drives, or any other mass storage devices may be used. Inembodiments, the main memory 4304, removable storage media 4305, readonly memory 4306, and mass storage 4307 may each comprise of one or moreunits linked together physically or logically to operate as a singleunit.

Bus 4301 communicatively couples processor(s) 4302 with the othermemory, storage and communication blocks. Bus 4301 can be a PCI/PCI-X orSCSI based system bus depending on the storage devices used. Removablestorage media 4305 can be any kind of external hard-drives, such as, butnot limited to, floppy drives, IOMEGA® Zip Drives, Compact Disc—ReadOnly Memory (CD-ROM), Compact Disc—Re-Writable (CD-RW), Digital VideoDisk—Read Only Memory (DVD-ROM).

Embodiments can be implemented using a combination of one or moredesired programming languages such as, but not limited to, PHP, MySQL,ASP, HTML, Javascript, and Flash.

Although a single computing device is illustrated in FIG. 43,embodiments can be extended to one or more systems of computing devicesoperating in parallel to present questions to one or more users. Forexample, a first computing device can be dedicated to a first user whilea second computing device can be dedicated to a second user where thefirst and second computing devices perform processes such as the processillustrated in FIG. 36 in parallel. In addition, for any given user,various aspects of processes such as the process of FIG. 36 can each beexecuted on a different processor. Accordingly, the embodiments caninclude massive parallel computing systems to accommodate any desirednumber of users. Example parallel processors include, but are notlimited to, the NVIDIA GT 2000 processor and the Intel Xeon processor.As an example, the Xeon process may be used in IBM's System x3650servers or Fujitsu's Primergy RX600 S4 servers. For example, FIG. 44illustrates an example parallel processing system 4400. The parallelprocessing system includes a common bus 4402 configured to connect toone or more systems. The parallel processing system 4400 includescomputing devices 4404-4434, with each of the computing devicesconfigured to communicated with each other via a connection to thecommon bus 4402. In embodiments, each of the computing devices 4404-4434has a configuration identical to the computing device 4300 (FIG. 43).The parallel processing system 4400 further includes a host 4436 that isconnected to the bus 4402. In embodiments, the host is a serverconfigured to control computing devices 4404-4434. The parallelprocessing system 4400 may be configured to include any desired numberof computing devices.

Various modifications and additions can be made to the exemplaryembodiments discussed without departing from the scope of the presentinvention. For example, while the embodiments described above refer toparticular features, the scope of this invention also includesembodiments having different combinations of features and embodimentsthat do not include all of the above described features.

1. A computer-based system for assessment driven learning, the systemcomprising: a generator configured to present questions received from aprocessor to a question presentation device utilizing a question moduleand a question database having a plurality of questions, wherein theplurality of questions are associated with a stacking weight, whereinthe plurality of questions are ordered according to the stacking weightof each question, and wherein each question presented includes at leastone answer option and at least one associated correct response; aquestion property evaluator configured to utilize the question module todetermine a likelihood of randomly selecting the at least one associatedcorrect response, wherein the question property evaluator considers, foreach question presented, at least the number of answer options and anumber of answer options a user is instructed to select; a questionresponse receiver configured to receive, from a device associated withthe user, at least one response to each question presented utilizing acommunications module; and a question response evaluator configured to,utilizing an evaluation module and a user response history database foreach question presented: determine, when the at least one response isthe at least one associated correct response, a likelihood that the atleast one response is a false positive using the question propertyevaluator, a response history of the user, and a response history of oneor more other users, and adjust the stacking weight of each questionpresented within the question database depending at least upon thelikelihood that the at least one response to the question is a falsepositive.
 2. The computer-based system according to claim 1, wherein foreach question presented, the question property evaluator is furtherconfigured to, utilizing the question module and question database,determine the likelihood of randomly selecting the at least oneassociated correct response for each question presented using an inverseof a number of combinations of answer options the user is instructed toselect from a set including the number of answer options when the numberof answer options the user is instructed to select is less than thenumber of answer options, and an inverse of two raised to a power of thenumber of answer options when the number of answer options the user isinstructed to select is greater than or equal to the number of answeroptions, and the question response evaluator is further configured to,utilizing the evaluation module and the user response history database,upon determination that the at least one response is the at least oneassociated correct response, increase the likelihood that the at leastone response is a false positive upon an increase in the likelihood ofrandomly selecting the at least one correct response.
 3. Thecomputer-based system according to claim 1, wherein the questionresponse evaluator is further configured to, utilizing the evaluationmodule and the user response history database for each questionpresented, increase the likelihood that the at least one response is afalse positive upon an indication from the user response history thatthe at least one response is not the at least one associated correctresponse consecutively one or more times, and decrease the likelihoodthat the at least one response is a false positive upon an indicationfrom the user response history that the at least one response is the atleast one associated correct response consecutively one or more times.4. The computer-based system according to claim 1, wherein for eachquestion presented, the question property evaluator is furtherconfigured to, utilizing the question module and question database,determine an amount of time the user takes to submit the at least oneassociated correct response, and determine, from the response history ofthe one or more other users the amount of time the user takes to submitthe at least one associated correct response, an average response timeof the one or more other users and the user to submit the at least oneassociated correct response, and the question response evaluator isfurther configured to, utilizing the evaluation module and questiondatabase, increase the likelihood that the at least one response is afalse positive upon determination that the amount of time the user takesto submit the at least one associated correct response is at least onestandard deviation below the average response time of the one or moreother users and the user to answer to submit the at least one associatedcorrect response.
 5. The computer-based system according to claim 4,wherein for each question presented, upon determination that the atleast one response is not the at least one associated correct responseand the user response history indicates that the user previouslysubmitted the at least one associated correct response, the questionproperty evaluator is further configured to, utilizing the questionmodule and the user response history database, remove an amount of timethe user took to previously submit the at least one associated correctresponse from the average response time of the one or more other usersand the user to submit the at least one associated correct response. 6.The computer-based system according to claim 1, wherein upondetermination that the user is permitted to report a level of confidencefor each question presented, the question response evaluator is furtherconfigured to, utilizing the evaluation module, set the likelihood thatthe at least one response is a false positive to a value representativeof a definite false positive when the reported level of confidence isless than a certainty threshold.
 7. The computer-based system accordingto claim 1, wherein upon determination that the user is permitted toreport a level of confidence in the at least one response for eachquestion presented, the question response evaluator is furtherconfigured to, utilizing the evaluation module and user response historydatabase, determine a first certainty weight that decreases the stackingweight upon determination that the reported level of confidence ishigher than a certainty threshold and upon an indication from the userresponse history that the at least one response is the at least oneassociated correct response consecutively one or more times, determine asecond certainty weight that increases the stacking weight upondetermination that the reported level of confidence is lower than thecertainty threshold and upon an indication from the user responsehistory that the at least one response is the at least one associatedcorrect response consecutively one or more times, determine a thirdcertainty weight that increases the stacking weight by an amount higherthan the second certainty weight upon determination that the reportedlevel of confidence is lower than the certainty threshold and upon anindication from the user response history that the at least one responseis not the at least one associated correct response consecutively one ormore times, and determine a fourth certainty weight that increases thestacking weight by an amount higher than the third certainty weight upondetermination that the reported level of confidence is higher than thecertainty threshold and upon an indication from the user responsehistory that the at least one response is not the at least oneassociated correct response consecutively one or more times.
 8. Thecomputer-based system according to claim 1, wherein for each questionpresented, the question property evaluator is further configured to,utilizing the question module and question database, determine alikelihood of selecting one or more incorrect options, wherein thequestion property evaluator considers at least a percentage of a numberof times that the one or more incorrect options are expected to beselected and a percentage of a number of times that the one or moreincorrect options are selected, and the question response evaluator isfurther configured to, utilizing the evaluation module, increase thelikelihood that the at least one response is a false positive upon adecrease in the likelihood of selecting the one or more incorrectoptions.
 9. A computer-implemented method for assessment drivenlearning, the method comprising: assigning a question stack to a user,wherein the question stack includes a plurality of questions; retrievingthe question stack from a question database; retrieving, by a questionproperty evaluator from the question database, one or more questionproperties associated with each question presented on a questionpresentation device, wherein each question presented includes at leastone answer option and at least on associated correct response, whereinthe one or more question properties indicate at least the number ofanswer options and a number of answer options the user is instructed toselect; receiving, utilizing a communications module, at least oneresponse to each question presented from a device associated with theuser; estimating, by the question property evaluator utilizing aquestion module for each question presented, a likelihood of randomlyselecting the at least one correct response using at least the number ofanswer options and the number of answer options the user is instructedto select; and adjusting, by the question response evaluator utilizingan evaluation module and a user response history database, an associatedstacking weight of each question presented depending at least upon alikelihood that the at least one response to the question is a falsepositive, wherein the plurality of questions are ordered according tothe associated stacking weight of each question presented, wherein, foreach question presented, when the at least one response to the questionis the at least one associated correct response, the question propertyevaluator estimates the likelihood that the at least one response is afalse positive using the question property evaluator, a response historyof the user, and a response history of one or more other users.
 10. Thecomputer-implemented method according to claim 9, further comprising:estimating, by the question property evaluator for each questionpresented, the likelihood of randomly selecting the at least oneassociated correct response using an inverse of a number of combinationsof answer options the user is instructed to select from a set of thenumber of answer options when the number of answer options the user isinstructed to select is less than the number of answer options, and aninverse of two raised to a power of the number of answer options whenthe number of answer options the user is instructed to select is greaterthan or equal to the number of answer options; and establishing, by thequestion response evaluator for each question presented upondetermination that the at least one response is the at least one correctresponse, a direct proportional relationship between the likelihood thatthe at least one response is a false positive and the likelihood ofrandomly selecting the at least one correct response, wherein, thequestion response evaluator increases the likelihood that the at leastone response is a false positive when the likelihood of randomlyselecting the at least one correct response increases.
 11. Thecomputer-implemented method of claim 9, further comprising: retrieving,for each question presented, the user response history from the userresponse history database; comparing, by the question response evaluatorfor each question presented, the at least one response to the retrieveduser response history database, wherein the question property evaluatorincreases the likelihood that the at least one response is a falsepositive upon an indication from the retrieved user response historythat the at least one response is not the at least one associatedcorrect response consecutively one or more times, and wherein thequestion property evaluator decreases the likelihood that the at leastone response is a false positive upon an indication from the userresponse history that the at least one response is the at least oneassociated correct response consecutively one or more times.
 12. Thecomputer-implemented method of claim 9, further comprising: retrieving,by the question property evaluator from the question database for eachquestion presented, an amount of time the user takes to submit the atleast one associated correct response and an average response time ofthe one or more other users to submit the at least one associatedcorrect response; adjusting, by the question property evaluator for eachquestion presented, the average response time of the one or more otherusers to submit the at least one associated correct response to includethat amount of time the user takes to submit the at least one associatedcorrect response; and increasing, by the question response propertyevaluator for each question presented, the likelihood that the at leastone response is a false positive upon determination that the amount oftime the user takes to submit the at least one associated correctresponse is at least one standard deviation below the adjusted averageresponse time to submit the at least one associated correct response.13. The computer-implemented method of claim 12, further comprising:retrieving, by the question property evaluator for each questionpresented upon determination that the at least one response is not theat least one associated response, the user response history from theuser response history database; and adjusting, by the question propertyevaluator utilizing the question module, the average response time ofthe one or more other users to submit the at least one associatedcorrect response upon indication from the retrieved user responsehistory that the user previously submitted the at least one associatedcorrect response, wherein the question property evaluator removes anamount of time the user took to previously submit the at least oneassociated correct response from the average response time of the one ormore other users to submit the at least one associated correct response.14. The computer-implemented method of claim 9, further comprising:receiving, from the device associated with the user, a reported level ofconfidence for each question presented, adjusting, by the questionresponse evaluator for each question presented upon determination thatthe received reported level of confidence is less than a certaintythreshold, the likelihood that the at least one response to the questionis a false positive to a value representative of a definite falsepositive.
 15. The computer-implemented method according to claim 9,further comprising: receiving, from the device associated with the user,a reported level of confidence for each question presented; retrieving,by the question response evaluator, the user response history from theuser response history database; and adjusting, by the question responseevaluator for each question presented, the stacking weight using thereceived reported level of confidence, wherein the question responseevaluator decreases the stacking weight by a first certainty weight upondetermination that the received reported level of confidence is higherthan a certainty threshold and upon an indication from the retrieveduser response history that the at least one response is the at least oneassociated correct response one or more times, the question responseevaluator increases the stacking weight by a second certainty weightupon determination that the reported level of confidence is lower thanthe certainty threshold and upon an indication from the retrieved userresponse history that the at least one response is the at least oneassociated correct response consecutively one or more times, thequestion response evaluator increases the stacking weight by a thirdcertainty weight that is an amount higher than the second certaintyweight upon determination that the reported level of confidence is lowerthan the certainty threshold and upon an indication from the retrieveduser response history that the at least one response is not the at leastone associated correct response consecutively one or more times, and thequestion response evaluator increases the stacking weight by an amounthigher than the third certainty weight upon determination that thereported level of confidence is higher than the certainty threshold andupon an indication from the retrieved user response history that the atleast one response is not the at least one associated correct responseconsecutively one or more times.
 16. The computer implemented methodaccording to claim 9, further comprising: estimating, by the questionproperty evaluator for each question presented, a likelihood ofselecting one or more incorrect options, wherein the question propertyevaluator considers at least a percentage of a number of times that theone or more incorrect options are expected to be selected and apercentage of a number of times that the one or more incorrect optionsare selected; and establishing an inverse relationship between thelikelihood that the at least one response is a false positive and thelikelihood of selecting the one or more incorrect options, wherein thequestion property evaluator increases the likelihood that the at leastone response is false positive when the likelihood of selecting the oneor more incorrect options decreases.
 17. A computer-readable mediumhaving instructions stored thereon, the instructions configured to causethe one or more processors to instruct: a generator to present questionsreceived from a processor to a question presentation device utilizing aquestion module and a question database having a plurality of questions,wherein the plurality of questions are associated with a stackingweight, wherein the plurality of questions are ordered according to thestacking weight of each question, and wherein each question presentedincludes at least one answer option and at least one associated correctresponse; a question property evaluator to utilize the question moduleto determine a likelihood of randomly selecting the at least oneassociated correct response, wherein the question property evaluatorconsiders, for each question presented, at least the number of answeroptions and a number of answer options a user is instructed to select; aquestion response receiver to receive, from a device associated with theuser, at least one response to each question presented utilizing acommunications module; and a question response evaluator to, utilizingan evaluation module and a user response history database for eachquestion presented: determine, when the at least one response is the atleast one associated correct response, a likelihood that the at leastone response is a false positive using the question property evaluator,a response history of the user, and a response history of one or moreother users, and adjust the stacking weight of each question presentedwithin the question database depending at least upon the likelihood thatthe at least one response to the question is a false positive.
 18. Thecomputer-readable medium according to claim 17, wherein the instructionsare further configured to cause the one or more processors to instruct:the question property evaluator to, utilizing the question module andquestion database for each question presented, determine the likelihoodof randomly selecting the at least one associated correct response foreach question presented using an inverse of a number of combinations ofanswer options the user is instructed to select from a set including thenumber of answer options when the number of answer options the user isinstructed to select is less than the number of answer options, and aninverse of two raised to a power of the number of answer options whenthe number of answer options the user is instructed to select is greaterthan or equal to the number of answer options, and the question responseevaluator to, utilizing the evaluation module and the user responsehistory database for each question presented, upon determination thatthe at least one response is the at least one associated correctresponse, increase the likelihood that the at least one response is afalse positive upon an increase in the likelihood of randomly selectingthe at least one correct response.
 19. The computer-readable mediumaccording to claim 17, wherein the instructions are further configuredto cause the one or more processors to instruct: the question responseevaluator to, utilizing the evaluation module and the user responsehistory database for each question presented, increase the likelihoodthat the at least one response is a false positive upon an indicationfrom the user response history that the at least one response is not theat least one associated correct response consecutively one or moretimes, and decrease the likelihood that the at least one response is afalse positive upon an indication from the user response history thatthe at least one response is the at least one associated correctresponse consecutively one or more times.
 20. The computer-readablemedium according to claim 17, wherein the instructions are furtherconfigured to cause the one or more processors to instruct: the questionproperty evaluator to, utilizing the question module and questiondatabase for each question presented, determine an amount of time theuser takes to submit the at least one associated correct response, anddetermine, from the response history of the one or more other users theamount of time the user takes to submit the at least one associatedcorrect response, an average response time of the one or more otherusers and the user to submit the at least one associated correctresponse, and the question response evaluator to, utilizing theevaluation module and question database for each question presented,increase the likelihood that the at least one response is a falsepositive upon determination that the amount of time the user takes tosubmit the at least one associated correct response is at least onestandard deviation below the average response time of the one or moreother users and the user to answer to submit the at least one associatedcorrect response.
 21. The computer-readable medium according to claim20, wherein the instructions are further configured to cause the one ormore processors to instruct: the question property evaluator to,utilizing the question module and the user response history database foreach question presented, remove an amount of time the user took topreviously submit the at least one associated correct response from theaverage response time of the one or more other users and the user tosubmit the at least one associated correct response upon determinationthat the at least one response is not the at least one associatedcorrect response and the user response history indicates that the userpreviously submitted the at least one associated correct response. 22.The computer-readable medium according to claim 17, wherein theinstructions are further configured to cause the one or more processorsto instruct: the question response evaluator to, utilizing theevaluation module upon determination that the user is permitted toreport a level of confidence for each question presented, set thelikelihood that the at least one response is a false positive to a valuerepresentative of a definite false positive when the reported level ofconfidence is less than a certainty threshold.
 23. The computer-readablemedium according to claim 17, wherein the instructions are furtherconfigured to cause the one or more processors to instruct: the questionresponse evaluator to, utilizing the evaluation module and user responsehistory database upon determination that the user is permitted to reporta level of confidence for each question presented, determine a firstcertainty weight that decreases the stacking weight upon determinationthat the reported level of confidence is higher than a certaintythreshold and upon an indication from the user response history that theat least one response is the at least on associated correct responseconsecutively one or more times, determine a second certainty weightthat increases the stacking weight upon determination that the reportedlevel of confidence is lower than the certainty threshold and upon anindication from the user response history that the at least one responseis the at least on associated correct response consecutively one or moretimes, determine a third certainty weight that increases the stackingweight by an amount higher than the second certainty weight upondetermination that the reported level of confidence is lower than thecertainty threshold and upon an indication from the user responsehistory that the at least one response is not the at least on associatedcorrect response consecutively one or more times, and determine a fourthcertainty weight that increases the stacking weight by an amount higherthan the third certainty weight upon determination that the reportedlevel of confidence is higher than the certainty threshold and upon anindication from the user response history that the at least one responseis not the at least on associated correct response consecutively one ormore times.
 24. The computer-readable medium according to claim 17,wherein the instructions are further configured to cause the one or moreprocessors to instruct: the question property evaluator to, utilizingthe question module and question database for each question presented,determine a likelihood of selecting one or more incorrect options,wherein the question property evaluator considers at least a percentageof a number of times that the one or more incorrect options are expectedto be selected and a percentage of a number of times that the one ormore incorrect options are selected, and the question response evaluatorto, utilizing the evaluation module for each question presented,increase the likelihood that the at least one response is a falsepositive upon a decrease in the likelihood of selecting the one or moreincorrect options.
 25. A system for assessment driven learning, thesystem comprising: means for displaying questions received from aprocessor on a question presentation device utilizing a questiondatabase having a plurality of questions, wherein the plurality ofquestion are associated with a stacking weight, wherein the plurality ofquestions are ordered according to the stacking weight of each question,and wherein each question presented includes at least two answer optionsand at least on associated correct response; means for determining, foreach question presented, a likelihood of randomly selecting the at leastone associated correct response using at least the number of answeroptions and a number of answer options a user is instructed to select;means for receiving, from a device associated with the user, at leastone response to each question presented; means for determining, for eachquestion presented, when the at least one response is the at least onassociated correct response, a likelihood that the at least one responseto the question is a false positive using the means for determining thelikelihood of randomly selecting the at least one associated correctresponse, a response history of the user, and a response historydatabase indicating a response history of one or more other users; andmeans for adjusting the stacking weight of the question depending atleast upon the likelihood that the at least one response to the questionis a false positive.
 26. The system according to claim 25, furthercomprising: means for determining, for each question presented, thelikelihood of randomly selecting the at least one associated correctresponse using an inverse of a number of combinations of answer optionsthe user is instructed to select from a set of the number of answeroptions when the number of answer options the user is instructed toselect is less than the number of answer options, and an inverse of tworaised to a power of the number of answer options when the number ofanswer options the user is instructed to select is greater than or equalto the number of answer options; and upon determination that the atleast one response to the question is the at least one correct response,means for increasing, for each question presented, the likelihood thatthe at least one response to the question is a false positive upon anincrease in the likelihood of randomly selecting the at least onecorrect response.
 27. The system according to claim 25, furthercomprising: means for increasing, for each question presented, thelikelihood that the at least one response to the question is a falsepositive upon an indication from the user response history that the atleast one response is not the at least one associated correct responseconsecutively one or more times; and means for decreasing, for eachquestion presented, the likelihood that the at least one response to thequestion is a false positive upon an indication from the user responsehistory that the at least one response is the at least one associatedcorrect response consecutively one or more times.
 28. The system ofclaim 25, further comprising: means for determining, for each questionpresented, an amount of time the user takes to submit the at least oneassociated correct response; means for determining, for each questionpresented, an average response time of the one or more other users andthe user to submit the at least one associated correct response usingthe response history of the one or more other users and the amount oftime the user takes to submit the at least one associated correctresponse; and means for increasing, the likelihood that the at least oneresponse is a false positive upon determination that the amount of timethe user takes to submit the at least one associated correct response isat least one standard deviation below the average response time of theone or more other users and the user to submit the at least oneassociated correct response.
 29. The system according to claim 25,further comprising: means for determining, upon determination that theuser is permitted to report a level of confidence for each questionpresented, a first certainty weight that decreases the stacking weightupon determination that the reported level of confidence is higher thana certainty threshold and upon an indication from the user responsehistory that the at least one response is the at least one associatedcorrect response one or more times, a second certainty weight thatincreases the stacking weight upon determination that the reported levelof confidence is lower than the certainty threshold and upon anindication from the user response history that the at least one responseis the at least one associated correct response consecutively one ormore times, a third certainty weight that increases the stacking weightby an amount higher than the second certainty weight upon determinationthat the reported level of confidence is lower than the certaintythreshold and upon an indication from the user response history that theat least one response is not the at least one associated responseconsecutively one or more times, and a fourth certainty weight thatincreases the stacking weight by an amount higher than the thirdcertainty weight upon determination that the reported level ofconfidence is higher than the certainty threshold and upon an indicationfrom the user response history that the at least one response is not theat least one associated correct response consecutively one or moretimes.
 30. The system according to claim 25, further comprising: meansfor determining, for each question presented, a likelihood of selectingone or more incorrect options using at least a percentage of a number oftimes that the one or more incorrect options are expected to be selectedand at least a percentage of a number of times that the one or moreincorrect options are selected; and means for increasing, for eachquestion presented, the likelihood that the at least one response to thequestion is a false positive upon a decrease in the likelihood ofselecting the one or more incorrect options.